2025-04-01 23:41:49,781 [ 538777 ] INFO : ClickHouse root is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse (runner:53, check_args_and_update_paths) 2025-04-01 23:41:49,781 [ 538777 ] INFO : Cases dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:97, check_args_and_update_paths) 2025-04-01 23:41:49,781 [ 538777 ] INFO : utils dir is not set. Will use /home/ubuntu/_work/ClickHouse/ClickHouse/utils (runner:108, check_args_and_update_paths) 2025-04-01 23:41:49,781 [ 538777 ] INFO : base_configs_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/programs/server, binary: /home/ubuntu/_work/_temp/test/build/clickhouse, cases_dir: /home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration (runner:110, check_args_and_update_paths) clickhouse_integration_tests_volume Running pytest container as: 'docker run --rm --name clickhouse_integration_tests_3ycre4 --privileged --dns-search='.' --memory=30709035008 --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=6712d5cc610d -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=caad4729259e -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 'test_refreshable_mat_view_replicated/test.py::test_alters[settings0-test_db-True-depends_on1-True-True]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-False-False]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-False-True]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-True-False]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-True-True]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-False-False]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-False-True]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-True-False]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-True-True]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-False-False]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-False-True]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-True-False]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-True-True]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-False-False]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-False-True]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-True-False]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-True-True]' 'test_refreshable_mat_view_replicated/test.py::test_append[False-False-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0]' 'test_refreshable_mat_view_replicated/test.py::test_append[False-False-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0]' 'test_refreshable_mat_view_replicated/test.py::test_append[False-True-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0]' 'test_refreshable_mat_view_replicated/test.py::test_append[False-True-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0]' 'test_refreshable_mat_view_replicated/test.py::test_append[True-False-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0]' 'test_refreshable_mat_view_replicated/test.py::test_append[True-False-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0]' 'test_refreshable_mat_view_replicated/test.py::test_append[True-True-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0]' 'test_refreshable_mat_view_replicated/test.py::test_append[True-True-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0]' test_refreshable_mat_view_replicated/test.py::test_long_query_cancel test_refreshable_mat_view_replicated/test.py::test_query_fail test_refreshable_mat_view_replicated/test.py::test_query_retry 'test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-False-False]' 'test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-False-True]' 'test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-True-False]' 'test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-True-True]' 'test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-False-False]' 'test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-False-True]' 'test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-True-False]' 'test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-True-True]' test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids test_replicated_merge_tree_encryption_codec/test.py::test_different_keys test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces test_replicated_merge_tree_encryption_codec/test.py::test_same_keys test_replicated_merge_tree_s3_zero_copy/test.py::test_drop_table 'test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[0-16]' 'test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[8192-12]' test_replicated_merge_tree_s3_zero_copy/test.py::test_s3_check_restore test_replicated_mutations/test.py::test_mutations 'test_replicated_mutations/test.py::test_mutations_dont_prevent_merges[nodes0]' 'test_replicated_mutations/test.py::test_mutations_dont_prevent_merges[nodes1]' test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach test_replicated_table_attach/test.py::test_startup_with_small_bg_pool test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned 'test_replicated_users/test.py::test_create_and_delete_replicated[QUOTA]' 'test_replicated_users/test.py::test_create_and_delete_replicated[ROLE]' 'test_replicated_users/test.py::test_create_and_delete_replicated[ROW_POLICY]' 'test_replicated_users/test.py::test_create_and_delete_replicated[SETTINGS_PROFILE]' 'test_replicated_users/test.py::test_create_and_delete_replicated[USER]' 'test_replicated_users/test.py::test_create_replicated[QUOTA]' 'test_replicated_users/test.py::test_create_replicated[ROLE]' 'test_replicated_users/test.py::test_create_replicated[ROW_POLICY]' 'test_replicated_users/test.py::test_create_replicated[SETTINGS_PROFILE]' 'test_replicated_users/test.py::test_create_replicated[USER]' 'test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[QUOTA]' 'test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[ROLE]' 'test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[ROW_POLICY]' 'test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[SETTINGS_PROFILE]' 'test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[USER]' 'test_replicated_users/test.py::test_create_replicated_on_cluster[QUOTA]' 'test_replicated_users/test.py::test_create_replicated_on_cluster[ROLE]' 'test_replicated_users/test.py::test_create_replicated_on_cluster[ROW_POLICY]' 'test_replicated_users/test.py::test_create_replicated_on_cluster[SETTINGS_PROFILE]' 'test_replicated_users/test.py::test_create_replicated_on_cluster[USER]' 'test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[QUOTA]' 'test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[ROLE]' 'test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[ROW_POLICY]' 'test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[SETTINGS_PROFILE]' 'test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[USER]' 'test_replicated_users/test.py::test_grant_revoke_replicated[With_ignored_on_cluster]' 'test_replicated_users/test.py::test_grant_revoke_replicated[Without_on_cluster]' test_replicated_users/test.py::test_reload_zookeeper 'test_replicated_users/test.py::test_rename_replicated[QUOTA]' 'test_replicated_users/test.py::test_rename_replicated[ROLE]' 'test_replicated_users/test.py::test_rename_replicated[ROW_POLICY]' 'test_replicated_users/test.py::test_rename_replicated[SETTINGS_PROFILE]' 'test_replicated_users/test.py::test_rename_replicated[USER]' test_s3_aws_sdk_has_slightly_unreliable_behaviour/test.py::test_dataloss test_s3_cluster/test.py::test_ambiguous_join test_s3_cluster/test.py::test_cluster_default_expression test_s3_cluster/test.py::test_cluster_format_detection test_s3_cluster/test.py::test_cluster_with_header test_s3_cluster/test.py::test_cluster_with_named_collection test_s3_cluster/test.py::test_count test_s3_cluster/test.py::test_count_macro test_s3_cluster/test.py::test_distributed_insert_select_with_replicated test_s3_cluster/test.py::test_distributed_s3_table_engine test_s3_cluster/test.py::test_hive_partitioning test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference test_s3_cluster/test.py::test_remote_hedged test_s3_cluster/test.py::test_remote_no_hedged test_s3_cluster/test.py::test_select_all test_s3_cluster/test.py::test_skip_unavailable_shards test_s3_cluster/test.py::test_union_all -vvv" altinityinfra/integration-tests-runner:cd6390247eca '. Start tests ============================= test session starts ============================== platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0 -- /usr/bin/python3 cachedir: .pytest_cache rootdir: /ClickHouse/tests/integration configfile: pytest.ini plugins: random-0.2, timeout-2.2.0, repeat-0.9.3, order-1.0.0, reportlog-0.4.0, xdist-3.5.0 timeout: 900.0s timeout method: signal timeout func_only: False created: 10/10 workers 10 workers [100 items] scheduling tests via LoadFileScheduling test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach test_replicated_mutations/test.py::test_mutations test_replicated_merge_tree_s3_zero_copy/test.py::test_drop_table test_refreshable_mat_view_replicated/test.py::test_alters[settings0-test_db-True-depends_on1-True-True] test_s3_cluster/test.py::test_ambiguous_join test_s3_aws_sdk_has_slightly_unreliable_behaviour/test.py::test_dataloss test_replicated_table_attach/test.py::test_startup_with_small_bg_pool test_replicated_users/test.py::test_create_and_delete_replicated[QUOTA] [gw4] [ 1%] PASSED test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach [gw2] [ 2%] PASSED test_replicated_users/test.py::test_create_and_delete_replicated[QUOTA] test_replicated_users/test.py::test_create_and_delete_replicated[ROLE] [gw2] [ 3%] PASSED test_replicated_users/test.py::test_create_and_delete_replicated[ROLE] test_replicated_users/test.py::test_create_and_delete_replicated[ROW_POLICY] [gw2] [ 4%] PASSED test_replicated_users/test.py::test_create_and_delete_replicated[ROW_POLICY] test_replicated_users/test.py::test_create_and_delete_replicated[SETTINGS_PROFILE] [gw2] [ 5%] PASSED test_replicated_users/test.py::test_create_and_delete_replicated[SETTINGS_PROFILE] test_replicated_users/test.py::test_create_and_delete_replicated[USER] [gw2] [ 6%] PASSED test_replicated_users/test.py::test_create_and_delete_replicated[USER] test_replicated_users/test.py::test_create_replicated[QUOTA] [gw2] [ 7%] PASSED test_replicated_users/test.py::test_create_replicated[QUOTA] test_replicated_users/test.py::test_create_replicated[ROLE] [gw1] [ 8%] PASSED test_s3_cluster/test.py::test_ambiguous_join test_s3_cluster/test.py::test_cluster_default_expression [gw5] [ 9%] PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids test_replicated_merge_tree_encryption_codec/test.py::test_different_keys [gw2] [ 10%] PASSED test_replicated_users/test.py::test_create_replicated[ROLE] test_replicated_users/test.py::test_create_replicated[ROW_POLICY] [gw2] [ 11%] PASSED test_replicated_users/test.py::test_create_replicated[ROW_POLICY] test_replicated_users/test.py::test_create_replicated[SETTINGS_PROFILE] [gw2] [ 12%] PASSED test_replicated_users/test.py::test_create_replicated[SETTINGS_PROFILE] test_replicated_users/test.py::test_create_replicated[USER] [gw5] [ 13%] PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_keys test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces [gw2] [ 14%] PASSED test_replicated_users/test.py::test_create_replicated[USER] test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[QUOTA] [gw1] [ 15%] PASSED test_s3_cluster/test.py::test_cluster_default_expression test_s3_cluster/test.py::test_cluster_format_detection [gw2] [ 16%] PASSED test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[QUOTA] test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[ROLE] [gw2] [ 17%] PASSED test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[ROLE] test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[ROW_POLICY] [gw2] [ 18%] PASSED test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[ROW_POLICY] test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[SETTINGS_PROFILE] [gw3] [ 19%] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings0-test_db-True-depends_on1-True-True] test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-False-False] [gw2] [ 20%] PASSED test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[SETTINGS_PROFILE] test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[USER] [gw2] [ 21%] PASSED test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[USER] test_replicated_users/test.py::test_create_replicated_on_cluster[QUOTA] [gw7] [ 22%] PASSED test_replicated_table_attach/test.py::test_startup_with_small_bg_pool test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned [gw2] [ 23%] PASSED test_replicated_users/test.py::test_create_replicated_on_cluster[QUOTA] test_replicated_users/test.py::test_create_replicated_on_cluster[ROLE] [gw2] [ 24%] PASSED test_replicated_users/test.py::test_create_replicated_on_cluster[ROLE] test_replicated_users/test.py::test_create_replicated_on_cluster[ROW_POLICY] [gw2] [ 25%] PASSED test_replicated_users/test.py::test_create_replicated_on_cluster[ROW_POLICY] test_replicated_users/test.py::test_create_replicated_on_cluster[SETTINGS_PROFILE] [gw5] [ 26%] PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces test_replicated_merge_tree_encryption_codec/test.py::test_same_keys [gw1] [ 27%] PASSED test_s3_cluster/test.py::test_cluster_format_detection test_s3_cluster/test.py::test_cluster_with_header [gw2] [ 28%] PASSED test_replicated_users/test.py::test_create_replicated_on_cluster[SETTINGS_PROFILE] test_replicated_users/test.py::test_create_replicated_on_cluster[USER] [gw2] [ 29%] PASSED test_replicated_users/test.py::test_create_replicated_on_cluster[USER] test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[QUOTA] [gw1] [ 30%] PASSED test_s3_cluster/test.py::test_cluster_with_header test_s3_cluster/test.py::test_cluster_with_named_collection [gw3] [ 31%] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-False-False] test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-False-True] [gw2] [ 32%] PASSED test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[QUOTA] test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[ROLE] [gw1] [ 33%] PASSED test_s3_cluster/test.py::test_cluster_with_named_collection test_s3_cluster/test.py::test_count [gw2] [ 34%] PASSED test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[ROLE] test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[ROW_POLICY] [gw1] [ 35%] PASSED test_s3_cluster/test.py::test_count test_s3_cluster/test.py::test_count_macro [gw5] [ 36%] PASSED test_replicated_merge_tree_encryption_codec/test.py::test_same_keys [gw2] [ 37%] PASSED test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[ROW_POLICY] test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[SETTINGS_PROFILE] [gw1] [ 38%] PASSED test_s3_cluster/test.py::test_count_macro test_s3_cluster/test.py::test_distributed_insert_select_with_replicated [gw2] [ 39%] PASSED test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[SETTINGS_PROFILE] test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[USER] [gw3] [ 40%] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-False-True] test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-True-False] [gw2] [ 41%] PASSED test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[USER] test_replicated_users/test.py::test_grant_revoke_replicated[With_ignored_on_cluster] [gw2] [ 42%] PASSED test_replicated_users/test.py::test_grant_revoke_replicated[With_ignored_on_cluster] test_replicated_users/test.py::test_grant_revoke_replicated[Without_on_cluster] [gw2] [ 43%] PASSED test_replicated_users/test.py::test_grant_revoke_replicated[Without_on_cluster] test_replicated_users/test.py::test_reload_zookeeper [gw3] [ 44%] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-True-False] test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-True-True] [gw3] [ 45%] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-True-True] test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-False-False] [gw6] [ 46%] PASSED test_replicated_mutations/test.py::test_mutations test_replicated_mutations/test.py::test_mutations_dont_prevent_merges[nodes0] [gw1] [ 47%] FAILED test_s3_cluster/test.py::test_distributed_insert_select_with_replicated test_s3_cluster/test.py::test_distributed_s3_table_engine [gw3] [ 48%] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-False-False] test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-False-True] [gw3] [ 49%] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-False-True] test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-True-False] [gw1] [ 50%] FAILED test_s3_cluster/test.py::test_distributed_s3_table_engine test_s3_cluster/test.py::test_hive_partitioning [gw3] [ 51%] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-True-False] test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-True-True] [gw7] [ 52%] PASSED test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned [gw3] [ 53%] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-True-True] test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-False-False] [gw3] [ 54%] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-False-False] test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-False-True] [gw1] [ 55%] FAILED test_s3_cluster/test.py::test_hive_partitioning test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference [gw3] [ 56%] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-False-True] test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-True-False] [gw3] [ 57%] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-True-False] test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-True-True] [gw1] [ 58%] FAILED test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference test_s3_cluster/test.py::test_remote_hedged [gw2] [ 59%] PASSED test_replicated_users/test.py::test_reload_zookeeper test_replicated_users/test.py::test_rename_replicated[QUOTA] [gw3] [ 60%] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-True-True] test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-False-False] [gw2] [ 61%] PASSED test_replicated_users/test.py::test_rename_replicated[QUOTA] test_replicated_users/test.py::test_rename_replicated[ROLE] [gw2] [ 62%] PASSED test_replicated_users/test.py::test_rename_replicated[ROLE] test_replicated_users/test.py::test_rename_replicated[ROW_POLICY] [gw2] [ 63%] PASSED test_replicated_users/test.py::test_rename_replicated[ROW_POLICY] test_replicated_users/test.py::test_rename_replicated[SETTINGS_PROFILE] [gw2] [ 64%] PASSED test_replicated_users/test.py::test_rename_replicated[SETTINGS_PROFILE] test_replicated_users/test.py::test_rename_replicated[USER] [gw2] [ 65%] PASSED test_replicated_users/test.py::test_rename_replicated[USER] [gw3] [ 66%] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-False-False] test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-False-True] [gw1] [ 67%] FAILED test_s3_cluster/test.py::test_remote_hedged test_s3_cluster/test.py::test_remote_no_hedged [gw3] [ 68%] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-False-True] test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-True-False] [gw1] [ 69%] FAILED test_s3_cluster/test.py::test_remote_no_hedged test_s3_cluster/test.py::test_select_all [gw6] [ 70%] PASSED test_replicated_mutations/test.py::test_mutations_dont_prevent_merges[nodes0] test_replicated_mutations/test.py::test_mutations_dont_prevent_merges[nodes1] [gw1] [ 71%] FAILED test_s3_cluster/test.py::test_select_all test_s3_cluster/test.py::test_skip_unavailable_shards [gw3] [ 72%] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-True-False] test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-True-True] [gw1] [ 73%] FAILED test_s3_cluster/test.py::test_skip_unavailable_shards test_s3_cluster/test.py::test_union_all [gw3] [ 74%] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-True-True] test_refreshable_mat_view_replicated/test.py::test_append[False-False-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] [gw1] [ 75%] FAILED test_s3_cluster/test.py::test_union_all [gw3] [ 76%] PASSED test_refreshable_mat_view_replicated/test.py::test_append[False-False-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] test_refreshable_mat_view_replicated/test.py::test_append[False-False-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] [gw3] [ 77%] PASSED test_refreshable_mat_view_replicated/test.py::test_append[False-False-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] test_refreshable_mat_view_replicated/test.py::test_append[False-True-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] [gw3] [ 78%] PASSED test_refreshable_mat_view_replicated/test.py::test_append[False-True-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] test_refreshable_mat_view_replicated/test.py::test_append[False-True-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] [gw3] [ 79%] PASSED test_refreshable_mat_view_replicated/test.py::test_append[False-True-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] test_refreshable_mat_view_replicated/test.py::test_append[True-False-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] [gw3] [ 80%] PASSED test_refreshable_mat_view_replicated/test.py::test_append[True-False-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] test_refreshable_mat_view_replicated/test.py::test_append[True-False-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] [gw3] [ 81%] PASSED test_refreshable_mat_view_replicated/test.py::test_append[True-False-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] test_refreshable_mat_view_replicated/test.py::test_append[True-True-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] [gw3] [ 82%] PASSED test_refreshable_mat_view_replicated/test.py::test_append[True-True-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] test_refreshable_mat_view_replicated/test.py::test_append[True-True-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] [gw3] [ 83%] PASSED test_refreshable_mat_view_replicated/test.py::test_append[True-True-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] test_refreshable_mat_view_replicated/test.py::test_long_query_cancel [gw3] [ 84%] SKIPPED test_refreshable_mat_view_replicated/test.py::test_long_query_cancel test_refreshable_mat_view_replicated/test.py::test_query_fail [gw3] [ 85%] SKIPPED test_refreshable_mat_view_replicated/test.py::test_query_fail test_refreshable_mat_view_replicated/test.py::test_query_retry [gw3] [ 86%] SKIPPED test_refreshable_mat_view_replicated/test.py::test_query_retry test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-False-False] [gw3] [ 87%] SKIPPED test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-False-False] test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-False-True] [gw3] [ 88%] SKIPPED test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-False-True] test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-True-False] [gw3] [ 89%] SKIPPED test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-True-False] test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-True-True] [gw3] [ 90%] SKIPPED test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-True-True] test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-False-False] [gw6] [ 91%] PASSED test_replicated_mutations/test.py::test_mutations_dont_prevent_merges[nodes1] [gw3] [ 92%] SKIPPED test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-False-False] test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-False-True] [gw3] [ 93%] SKIPPED test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-False-True] test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-True-False] [gw3] [ 94%] SKIPPED test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-True-False] test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-True-True] [gw3] [ 95%] SKIPPED test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-True-True] [gw8] [ 96%] PASSED test_s3_aws_sdk_has_slightly_unreliable_behaviour/test.py::test_dataloss [gw0] [ 97%] PASSED test_replicated_merge_tree_s3_zero_copy/test.py::test_drop_table test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[0-16] [gw0] [ 98%] PASSED test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[0-16] test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[8192-12] [gw0] [ 99%] PASSED test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[8192-12] test_replicated_merge_tree_s3_zero_copy/test.py::test_s3_check_restore [gw0] [100%] PASSED test_replicated_merge_tree_s3_zero_copy/test.py::test_s3_check_restore =================================== FAILURES =================================== ________________ test_distributed_insert_select_with_replicated ________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_distributed_insert_select_with_replicated(started_cluster): first_replica_first_shard = started_cluster.instances["s0_0_0"] second_replica_first_shard = started_cluster.instances["s0_0_1"] first_replica_first_shard.query( """DROP TABLE IF EXISTS insert_select_replicated_local ON CLUSTER 'first_shard' SYNC;""" ) first_replica_first_shard.query( """ CREATE TABLE insert_select_replicated_local ON CLUSTER 'first_shard' (a String, b UInt64) ENGINE=ReplicatedMergeTree('/clickhouse/tables/{shard}/insert_select_with_replicated', '{replica}') ORDER BY (a, b); """ ) for replica in [first_replica_first_shard, second_replica_first_shard]: replica.query( """ SYSTEM STOP FETCHES; """ ) replica.query( """ SYSTEM STOP MERGES; """ ) > first_replica_first_shard.query( """ INSERT INTO insert_select_replicated_local SELECT * FROM s3Cluster( 'first_shard', 'http://minio1:9001/root/data/generated/*.csv', 'minio', 'minio123', 'CSV','a String, b UInt64' ) SETTINGS parallel_distributed_insert_select=1; """ ) test_s3_cluster/test.py:393: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3678: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 32, stderr: [s0_0_0] 2025.04.01 23:42:39.513205 [ 686 ] {f1cdec9f-8973-4b17-b5a1-a89871a1cb35} : Logical error: 'Replica info is not initialized'. E [s0_0_0] 2025.04.01 23:42:39.543911 [ 686 ] {f1cdec9f-8973-4b17-b5a1-a89871a1cb35} : Stack trace (when copying this message, always include the lines below): E E 0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x0000000038031254 E 1. ./build_docker/./src/Common/Exception.cpp:105: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001bc0ed05 E 2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000b291545 E 3. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000b2addce E 4. ./build_docker/./src/QueryPipeline/RemoteQueryExecutor.cpp:727: DB::RemoteQueryExecutor::processReadTaskRequest() @ 0x0000000029aed2bb E 5. ./build_docker/./src/QueryPipeline/RemoteQueryExecutor.cpp:623: DB::RemoteQueryExecutor::processPacket(DB::Packet) @ 0x0000000029ae7b24 E 6. ./build_docker/./src/QueryPipeline/RemoteQueryExecutor.cpp:562: DB::RemoteQueryExecutor::readAsync() @ 0x0000000029aeb3eb E 7. ./build_docker/./src/Processors/Sources/RemoteSource.cpp:182: DB::RemoteSource::tryGenerate() @ 0x00000000315014a7 E 8. ./build_docker/./src/Processors/ISource.cpp:108: DB::ISource::work() @ 0x0000000030c354d7 E 9. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:49: DB::ExecutionThreadContext::executeTask() @ 0x0000000030c711ce E 10. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:290: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x0000000030c55e51 E 11. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:256: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x0000000030c543fc E 12. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:127: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000030c53ee2 E 13. ./build_docker/./src/Processors/Executors/CompletedPipelineExecutor.cpp:49: void std::__function::__policy_invoker::__call_impl::ThreadFromGlobalPoolImpl(DB::CompletedPipelineExecutor::execute()::$_0&&)::'lambda'(), void ()>>(std::__function::__policy_storage const*) @ 0x0000000030c51f07 E 14. ./contrib/llvm-project/libcxx/include/__functional/function.h:848: ? @ 0x000000001bdda21b E 15. ./contrib/llvm-project/libcxx/include/__functional/invoke.h:359: ? @ 0x000000001bde74f0 E 16. asan_thread_start(void*) @ 0x000000000b240e77 E 17. ? @ 0x00007fbdd0475ac3 E 18. ? @ 0x00007fbdd0507850 E E [s0_0_0] 2025.04.01 23:42:39.545322 [ 737 ] BaseDaemon: ######################################## E [s0_0_0] 2025.04.01 23:42:39.545458 [ 737 ] BaseDaemon: (version 24.12.2.20221.altinityantalya (altinity build), build id: E3F43B0C9BFE6311FC1B0D8F77858862995FA832, git hash: 82252d159dc02cab0f366aaa5691adc1545dd11d) (from thread 686) (query_id: f1cdec9f-8973-4b17-b5a1-a89871a1cb35) (query: INSERT INTO insert_select_replicated_local SETTINGS parallel_distributed_insert_select = 1 SELECT * FROM s3Cluster('first_shard', 'http://minio1:9001/root/data/generated/*.csv', 'minio', '[HIDDEN]', 'CSV', 'a String, b UInt64') SETTINGS parallel_distributed_insert_select = 1) Received signal Aborted (6) E [s0_0_0] 2025.04.01 23:42:39.545560 [ 737 ] BaseDaemon: E [s0_0_0] 2025.04.01 23:42:39.545622 [ 737 ] BaseDaemon: Stack trace: 0x000056287e52b32d 0x000056287eb89aba 0x00007fbdd0423520 0x00007fbdd04779fd 0x00007fbdd0423476 0x00007fbdd04097f3 0x000056287e4afc6e 0x000056287e4b121c 0x000056286db33545 0x000056286db4fdce 0x000056288c38f2bb 0x000056288c389b24 0x000056288c38d3eb 0x0000562893da34a7 0x00005628934d74d7 0x00005628935131ce 0x00005628934f7e51 0x00005628934f63fc 0x00005628934f5ee2 0x00005628934f3f07 0x000056287e67c21b 0x000056287e6894f0 0x000056286dae2e77 0x00007fbdd0475ac3 0x00007fbdd0507850 E [s0_0_0] 2025.04.01 23:42:39.740196 [ 737 ] BaseDaemon: 0.0. inlined from ./build_docker/./src/Common/StackTrace.cpp:381: StackTrace::tryCapture() E [s0_0_0] 2025.04.01 23:42:39.740328 [ 737 ] BaseDaemon: 0. ./build_docker/./src/Common/StackTrace.cpp:350: StackTrace::StackTrace(ucontext_t const&) @ 0x000000001bc8932d E [s0_0_0] 2025.04.01 23:42:39.837556 [ 737 ] BaseDaemon: 1. ./build_docker/./src/Common/SignalHandlers.cpp:102: signalHandler(int, siginfo_t*, void*) @ 0x000000001c2e7aba E [s0_0_0] 2025.04.01 23:42:39.837668 [ 737 ] BaseDaemon: 2. ? @ 0x00007fbdd0423520 E [s0_0_0] 2025.04.01 23:42:39.837703 [ 737 ] BaseDaemon: 3. ? @ 0x00007fbdd04779fd E [s0_0_0] 2025.04.01 23:42:39.837745 [ 737 ] BaseDaemon: 4. ? @ 0x00007fbdd0423476 E [s0_0_0] 2025.04.01 23:42:39.837780 [ 737 ] BaseDaemon: 5. ? @ 0x00007fbdd04097f3 E [s0_0_0] 2025.04.01 23:42:39.914845 [ 737 ] BaseDaemon: 6. ./build_docker/./src/Common/Exception.cpp:48: DB::abortOnFailedAssertion(String const&, void* const*, unsigned long, unsigned long) @ 0x000000001bc0dc6e E [s0_0_0] 2025.04.01 23:42:39.986674 [ 737 ] BaseDaemon: 7.0. inlined from ./build_docker/./src/Common/Exception.cpp:70: DB::handle_error_code(String const&, int, bool, std::vector> const&) E [s0_0_0] 2025.04.01 23:42:39.986800 [ 737 ] BaseDaemon: 7. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001bc0f21c E [s0_0_0] 2025.04.01 23:42:44.257108 [ 737 ] BaseDaemon: 8. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000b291545 E [s0_0_0] 2025.04.01 23:42:44.315997 [ 737 ] BaseDaemon: 9. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000b2addce E [s0_0_0] 2025.04.01 23:42:44.680506 [ 737 ] BaseDaemon: 10. ./build_docker/./src/QueryPipeline/RemoteQueryExecutor.cpp:727: DB::RemoteQueryExecutor::processReadTaskRequest() @ 0x0000000029aed2bb E [s0_0_0] 2025.04.01 23:42:44.856580 [ 737 ] BaseDaemon: 11. ./build_docker/./src/QueryPipeline/RemoteQueryExecutor.cpp:623: DB::RemoteQueryExecutor::processPacket(DB::Packet) @ 0x0000000029ae7b24 E [s0_0_0] 2025.04.01 23:42:45.069846 [ 737 ] BaseDaemon: 12. ./build_docker/./src/QueryPipeline/RemoteQueryExecutor.cpp:562: DB::RemoteQueryExecutor::readAsync() @ 0x0000000029aeb3eb E [s0_0_0] 2025.04.01 23:42:45.157672 [ 737 ] BaseDaemon: 13. ./build_docker/./src/Processors/Sources/RemoteSource.cpp:182: DB::RemoteSource::tryGenerate() @ 0x00000000315014a7 E [s0_0_0] 2025.04.01 23:42:45.197092 [ 737 ] BaseDaemon: 14. ./build_docker/./src/Processors/ISource.cpp:108: DB::ISource::work() @ 0x0000000030c354d7 E [s0_0_0] 2025.04.01 23:42:45.221630 [ 737 ] BaseDaemon: 15.0. inlined from ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:49: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) E [s0_0_0] 2025.04.01 23:42:45.221766 [ 737 ] BaseDaemon: 15. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:98: DB::ExecutionThreadContext::executeTask() @ 0x0000000030c711ce E [s0_0_0] 2025.04.01 23:42:45.274318 [ 737 ] BaseDaemon: 16. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:290: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x0000000030c55e51 E [s0_0_0] 2025.04.01 23:42:45.318842 [ 737 ] BaseDaemon: 17.0. inlined from ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:256: DB::PipelineExecutor::executeSingleThread(unsigned long) E [s0_0_0] 2025.04.01 23:42:45.318948 [ 737 ] BaseDaemon: 17. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:442: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x0000000030c543fc E [s0_0_0] 2025.04.01 23:42:45.375401 [ 737 ] BaseDaemon: 18. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:127: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000030c53ee2 E [s0_0_0] 2025.04.01 23:42:45.402272 [ 737 ] BaseDaemon: 19.0. inlined from ./build_docker/./src/Processors/Executors/CompletedPipelineExecutor.cpp:49: DB::threadFunction(DB::CompletedPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) E [s0_0_0] 2025.04.01 23:42:45.402500 [ 737 ] BaseDaemon: 19.1. inlined from ./build_docker/./src/Processors/Executors/CompletedPipelineExecutor.cpp:89: operator() E [s0_0_0] 2025.04.01 23:42:45.402659 [ 737 ] BaseDaemon: 19.2. inlined from ./contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? E [s0_0_0] 2025.04.01 23:42:45.402840 [ 737 ] BaseDaemon: 19.3. inlined from ./contrib/llvm-project/libcxx/include/tuple:1789: _ZNSt3__118__apply_tuple_implB6v15007IRZN2DB25CompletedPipelineExecutor7executeEvE3$_0RNS_5tupleIJEEETpTnmJEEEDcOT_OT0_NS_15__tuple_indicesIJXspT1_EEEE E [s0_0_0] 2025.04.01 23:42:45.403077 [ 737 ] BaseDaemon: 19.4. inlined from ./contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15007]&>(DB::CompletedPipelineExecutor::execute()::$_0&, std::tuple<>&) E [s0_0_0] 2025.04.01 23:42:45.403290 [ 737 ] BaseDaemon: 19.5. inlined from ./src/Common/ThreadPool.h:311: operator() E [s0_0_0] 2025.04.01 23:42:45.403491 [ 737 ] BaseDaemon: 19.6. inlined from ./contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? E [s0_0_0] 2025.04.01 23:42:45.403651 [ 737 ] BaseDaemon: 19.7. inlined from ./contrib/llvm-project/libcxx/include/__functional/invoke.h:479: ? E [s0_0_0] 2025.04.01 23:42:45.403807 [ 737 ] BaseDaemon: 19.8. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:235: ? E [s0_0_0] 2025.04.01 23:42:45.403986 [ 737 ] BaseDaemon: 19. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000030c51f07 E [s0_0_0] 2025.04.01 23:42:45.446414 [ 737 ] BaseDaemon: 20.0. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:848: ? E [s0_0_0] 2025.04.01 23:42:45.446578 [ 737 ] BaseDaemon: 20.1. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:1197: ? E [s0_0_0] 2025.04.01 23:42:45.446628 [ 737 ] BaseDaemon: 20. ./build_docker/./src/Common/ThreadPool.cpp:785: ThreadPoolImpl::ThreadFromThreadPool::worker() @ 0x000000001bdda21b E [s0_0_0] 2025.04.01 23:42:45.516875 [ 737 ] BaseDaemon: 21.0. inlined from ./contrib/llvm-project/libcxx/include/__functional/invoke.h:359: ? E [s0_0_0] 2025.04.01 23:42:45.517014 [ 737 ] BaseDaemon: 21.1. inlined from ./contrib/llvm-project/libcxx/include/thread:284: void std::__thread_execute[abi:v15007]>, void (ThreadPoolImpl::ThreadFromThreadPool::*)(), ThreadPoolImpl::ThreadFromThreadPool*, 2ul>(std::tuple>, void (ThreadPoolImpl::ThreadFromThreadPool::*)(), ThreadPoolImpl::ThreadFromThreadPool*>&, std::__tuple_indices<2ul>) E [s0_0_0] 2025.04.01 23:42:45.517074 [ 737 ] BaseDaemon: 21. ./contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15007]>, void (ThreadPoolImpl::ThreadFromThreadPool::*)(), ThreadPoolImpl::ThreadFromThreadPool*>>(void*) @ 0x000000001bde74f0 E [s0_0_0] 2025.04.01 23:42:45.550499 [ 737 ] BaseDaemon: 22. asan_thread_start(void*) @ 0x000000000b240e77 E [s0_0_0] 2025.04.01 23:42:45.550625 [ 737 ] BaseDaemon: 23. ? @ 0x00007fbdd0475ac3 E [s0_0_0] 2025.04.01 23:42:45.550667 [ 737 ] BaseDaemon: 24. ? @ 0x00007fbdd0507850 E [s0_0_0] 2025.04.01 23:42:46.052060 [ 737 ] BaseDaemon: Integrity check of the executable successfully passed (checksum: DB310373A4AD542A55A473D2E57E140B) E [s0_0_0] 2025.04.01 23:42:47.740685 [ 737 ] BaseDaemon: This ClickHouse version is not official and should be upgraded to the official build. E [s0_0_0] 2025.04.01 23:42:47.741070 [ 737 ] BaseDaemon: Changed settings: parallel_distributed_insert_select = 1 E [s0_0_0] 2025.04.01 23:42:51.547081 [ 738 ] BaseDaemon: ######################################## E [s0_0_0] 2025.04.01 23:42:51.547180 [ 738 ] BaseDaemon: (version 24.12.2.20221.altinityantalya (altinity build), build id: E3F43B0C9BFE6311FC1B0D8F77858862995FA832, git hash: 82252d159dc02cab0f366aaa5691adc1545dd11d) (from thread 686) (query_id: f1cdec9f-8973-4b17-b5a1-a89871a1cb35) (query: INSERT INTO insert_select_replicated_local SETTINGS parallel_distributed_insert_select = 1 SELECT * FROM s3Cluster('first_shard', 'http://minio1:9001/root/data/generated/*.csv', 'minio', '[HIDDEN]', 'CSV', 'a String, b UInt64') SETTINGS parallel_distributed_insert_select = 1) Received signal Segmentation fault (11) E [s0_0_0] 2025.04.01 23:42:51.547305 [ 738 ] BaseDaemon: Address: NULL pointer. Access: read. Unknown si_code. E [s0_0_0] 2025.04.01 23:42:51.547344 [ 738 ] BaseDaemon: Stack trace: 0x000056287e52b32d 0x000056287eb89aba 0x00007fbdd0423520 0x00007fbdd0409899 0x000056287e4afc6e 0x000056287e4b121c 0x000056286db33545 0x000056286db4fdce 0x000056288c38f2bb 0x000056288c389b24 0x000056288c38d3eb 0x0000562893da34a7 0x00005628934d74d7 0x00005628935131ce 0x00005628934f7e51 0x00005628934f63fc 0x00005628934f5ee2 0x00005628934f3f07 0x000056287e67c21b 0x000056287e6894f0 0x000056286dae2e77 0x00007fbdd0475ac3 0x00007fbdd0507850 E [s0_0_0] 2025.04.01 23:42:51.591285 [ 738 ] BaseDaemon: 0.0. inlined from ./build_docker/./src/Common/StackTrace.cpp:381: StackTrace::tryCapture() E [s0_0_0] 2025.04.01 23:42:51.591400 [ 738 ] BaseDaemon: 0. ./build_docker/./src/Common/StackTrace.cpp:350: StackTrace::StackTrace(ucontext_t const&) @ 0x000000001bc8932d E [s0_0_0] 2025.04.01 23:42:51.641664 [ 738 ] BaseDaemon: 1. ./build_docker/./src/Common/SignalHandlers.cpp:102: signalHandler(int, siginfo_t*, void*) @ 0x000000001c2e7aba E [s0_0_0] 2025.04.01 23:42:51.641815 [ 738 ] BaseDaemon: 2. ? @ 0x00007fbdd0423520 E [s0_0_0] 2025.04.01 23:42:51.641866 [ 738 ] BaseDaemon: 3. ? @ 0x00007fbdd0409899 E [s0_0_0] 2025.04.01 23:42:51.736198 [ 738 ] BaseDaemon: 4. ./build_docker/./src/Common/Exception.cpp:48: DB::abortOnFailedAssertion(String const&, void* const*, unsigned long, unsigned long) @ 0x000000001bc0dc6e E [s0_0_0] 2025.04.01 23:42:51.824352 [ 738 ] BaseDaemon: 5.0. inlined from ./build_docker/./src/Common/Exception.cpp:70: DB::handle_error_code(String const&, int, bool, std::vector> const&) E [s0_0_0] 2025.04.01 23:42:51.824517 [ 738 ] BaseDaemon: 5. ./build_docker/./src/Common/Exception.cpp:111: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001bc0f21c E [s0_0_0] 2025.04.01 23:42:51.867273 [ 738 ] BaseDaemon: 6. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000b291545 E [s0_0_0] 2025.04.01 23:42:51.905125 [ 738 ] BaseDaemon: 7. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000b2addce E [s0_0_0] 2025.04.01 23:42:52.115477 [ 738 ] BaseDaemon: 8. ./build_docker/./src/QueryPipeline/RemoteQueryExecutor.cpp:727: DB::RemoteQueryExecutor::processReadTaskRequest() @ 0x0000000029aed2bb E [s0_0_0] 2025.04.01 23:42:52.341438 [ 738 ] BaseDaemon: 9. ./build_docker/./src/QueryPipeline/RemoteQueryExecutor.cpp:623: DB::RemoteQueryExecutor::processPacket(DB::Packet) @ 0x0000000029ae7b24 E [s0_0_0] 2025.04.01 23:42:52.520348 [ 738 ] BaseDaemon: 10. ./build_docker/./src/QueryPipeline/RemoteQueryExecutor.cpp:562: DB::RemoteQueryExecutor::readAsync() @ 0x0000000029aeb3eb E [s0_0_0] 2025.04.01 23:42:52.572278 [ 738 ] BaseDaemon: 11. ./build_docker/./src/Processors/Sources/RemoteSource.cpp:182: DB::RemoteSource::tryGenerate() @ 0x00000000315014a7 E [s0_0_0] 2025.04.01 23:42:52.602783 [ 738 ] BaseDaemon: 12. ./build_docker/./src/Processors/ISource.cpp:108: DB::ISource::work() @ 0x0000000030c354d7 E [s0_0_0] 2025.04.01 23:42:52.618109 [ 738 ] BaseDaemon: 13.0. inlined from ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:49: DB::executeJob(DB::ExecutingGraph::Node*, DB::ReadProgressCallback*) E [s0_0_0] 2025.04.01 23:42:52.618252 [ 738 ] BaseDaemon: 13. ./build_docker/./src/Processors/Executors/ExecutionThreadContext.cpp:98: DB::ExecutionThreadContext::executeTask() @ 0x0000000030c711ce E [s0_0_0] 2025.04.01 23:42:52.665029 [ 738 ] BaseDaemon: 14. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:290: DB::PipelineExecutor::executeStepImpl(unsigned long, std::atomic*) @ 0x0000000030c55e51 E [s0_0_0] 2025.04.01 23:42:52.706112 [ 738 ] BaseDaemon: 15.0. inlined from ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:256: DB::PipelineExecutor::executeSingleThread(unsigned long) E [s0_0_0] 2025.04.01 23:42:52.706230 [ 738 ] BaseDaemon: 15. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:442: DB::PipelineExecutor::executeImpl(unsigned long, bool) @ 0x0000000030c543fc E [s0_0_0] 2025.04.01 23:42:52.755845 [ 738 ] BaseDaemon: 16. ./build_docker/./src/Processors/Executors/PipelineExecutor.cpp:127: DB::PipelineExecutor::execute(unsigned long, bool) @ 0x0000000030c53ee2 E [s0_0_0] 2025.04.01 23:42:52.775609 [ 738 ] BaseDaemon: 17.0. inlined from ./build_docker/./src/Processors/Executors/CompletedPipelineExecutor.cpp:49: DB::threadFunction(DB::CompletedPipelineExecutor::Data&, std::shared_ptr, unsigned long, bool) E [s0_0_0] 2025.04.01 23:42:52.775733 [ 738 ] BaseDaemon: 17.1. inlined from ./build_docker/./src/Processors/Executors/CompletedPipelineExecutor.cpp:89: operator() E [s0_0_0] 2025.04.01 23:42:52.775777 [ 738 ] BaseDaemon: 17.2. inlined from ./contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? E [s0_0_0] 2025.04.01 23:42:52.775826 [ 738 ] BaseDaemon: 17.3. inlined from ./contrib/llvm-project/libcxx/include/tuple:1789: _ZNSt3__118__apply_tuple_implB6v15007IRZN2DB25CompletedPipelineExecutor7executeEvE3$_0RNS_5tupleIJEEETpTnmJEEEDcOT_OT0_NS_15__tuple_indicesIJXspT1_EEEE E [s0_0_0] 2025.04.01 23:42:52.775887 [ 738 ] BaseDaemon: 17.4. inlined from ./contrib/llvm-project/libcxx/include/tuple:1798: decltype(auto) std::apply[abi:v15007]&>(DB::CompletedPipelineExecutor::execute()::$_0&, std::tuple<>&) E [s0_0_0] 2025.04.01 23:42:52.775975 [ 738 ] BaseDaemon: 17.5. inlined from ./src/Common/ThreadPool.h:311: operator() E [s0_0_0] 2025.04.01 23:42:52.776012 [ 738 ] BaseDaemon: 17.6. inlined from ./contrib/llvm-project/libcxx/include/__functional/invoke.h:394: ? E [s0_0_0] 2025.04.01 23:42:52.776064 [ 738 ] BaseDaemon: 17.7. inlined from ./contrib/llvm-project/libcxx/include/__functional/invoke.h:479: ? E [s0_0_0] 2025.04.01 23:42:52.776113 [ 738 ] BaseDaemon: 17.8. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:235: ? E [s0_0_0] 2025.04.01 23:42:52.776177 [ 738 ] BaseDaemon: 17. ./contrib/llvm-project/libcxx/include/__functional/function.h:716: ? @ 0x0000000030c51f07 E [s0_0_0] 2025.04.01 23:42:52.811082 [ 738 ] BaseDaemon: 18.0. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:848: ? E [s0_0_0] 2025.04.01 23:42:52.811204 [ 738 ] BaseDaemon: 18.1. inlined from ./contrib/llvm-project/libcxx/include/__functional/function.h:1197: ? E [s0_0_0] 2025.04.01 23:42:52.811285 [ 738 ] BaseDaemon: 18. ./build_docker/./src/Common/ThreadPool.cpp:785: ThreadPoolImpl::ThreadFromThreadPool::worker() @ 0x000000001bdda21b E [s0_0_0] 2025.04.01 23:42:52.884165 [ 738 ] BaseDaemon: 19.0. inlined from ./contrib/llvm-project/libcxx/include/__functional/invoke.h:359: ? E [s0_0_0] 2025.04.01 23:42:52.884354 [ 738 ] BaseDaemon: 19.1. inlined from ./contrib/llvm-project/libcxx/include/thread:284: void std::__thread_execute[abi:v15007]>, void (ThreadPoolImpl::ThreadFromThreadPool::*)(), ThreadPoolImpl::ThreadFromThreadPool*, 2ul>(std::tuple>, void (ThreadPoolImpl::ThreadFromThreadPool::*)(), ThreadPoolImpl::ThreadFromThreadPool*>&, std::__tuple_indices<2ul>) E [s0_0_0] 2025.04.01 23:42:52.884475 [ 738 ] BaseDaemon: 19. ./contrib/llvm-project/libcxx/include/thread:295: void* std::__thread_proxy[abi:v15007]>, void (ThreadPoolImpl::ThreadFromThreadPool::*)(), ThreadPoolImpl::ThreadFromThreadPool*>>(void*) @ 0x000000001bde74f0 E [s0_0_0] 2025.04.01 23:42:52.919490 [ 738 ] BaseDaemon: 20. asan_thread_start(void*) @ 0x000000000b240e77 E [s0_0_0] 2025.04.01 23:42:52.919618 [ 738 ] BaseDaemon: 21. ? @ 0x00007fbdd0475ac3 E [s0_0_0] 2025.04.01 23:42:52.919694 [ 738 ] BaseDaemon: 22. ? @ 0x00007fbdd0507850 E [s0_0_0] 2025.04.01 23:42:53.393843 [ 738 ] BaseDaemon: Integrity check of the executable successfully passed (checksum: DB310373A4AD542A55A473D2E57E140B) E Error on processing query: Code: 32. DB::Exception: Attempt to read after eof: while receiving packet from 172.16.8.11:9000. (ATTEMPT_TO_READ_AFTER_EOF), Stack trace (when copying this message, always include the lines below): E E 0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x0000000038031254 E 1. ./build_docker/./src/Common/Exception.cpp:105: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001bc0ed05 E 2. DB::Exception::Exception(PreformattedMessage&&, int) @ 0x000000000b291545 E 3. DB::Exception::Exception<>(int, FormatStringHelperImpl<>) @ 0x000000000b2addce E 4. ./build_docker/./src/IO/VarInt.cpp:13: DB::throwReadAfterEOF() @ 0x000000001bd5bc0b E 5. ./src/IO/VarInt.h:78: void DB::varint_impl::readVarUInt(unsigned long&, DB::ReadBuffer&) @ 0x000000001bd083d0 E 6. ./src/IO/VarInt.h:96: DB::Connection::receivePacket() @ 0x00000000307f4ffa E 7. ./build_docker/./src/Client/ClientBase.cpp:1247: DB::ClientBase::receiveAndProcessPacket(std::shared_ptr, bool) @ 0x00000000307a2afe E 8. ./build_docker/./src/Client/ClientBase.cpp:1222: DB::ClientBase::receiveResult(std::shared_ptr, int, bool) @ 0x00000000307a1867 E 9. ./build_docker/./src/Client/ClientBase.cpp:1137: DB::ClientBase::processOrdinaryQuery(String const&, std::shared_ptr) @ 0x000000003079fbf1 E 10. ./build_docker/./src/Client/ClientBase.cpp:2095: DB::ClientBase::processParsedSingleQuery(String const&, String const&, std::shared_ptr, std::optional, bool) @ 0x000000003079bfc3 E 11. ./build_docker/./src/Client/ClientBase.cpp:2436: DB::ClientBase::executeMultiQuery(String const&) @ 0x00000000307b41f8 E 12. ./build_docker/./src/Client/ClientBase.cpp:2582: DB::ClientBase::processQueryText(String const&) @ 0x00000000307b602a E 13. ./build_docker/./src/Client/ClientBase.cpp:2938: DB::ClientBase::runNonInteractive() @ 0x00000000307bc57b E 14. ./build_docker/./programs/client/Client.cpp:405: DB::Client::main(std::vector> const&) @ 0x000000001bff459f E 15. ./build_docker/./base/poco/Util/src/Application.cpp:315: Poco::Util::Application::run() @ 0x00000000382716b7 E 16. ./build_docker/./programs/client/Client.cpp:1399: mainEntryClickHouseClient(int, char**) @ 0x000000001c010ec9 E 17. ./build_docker/./programs/main.cpp:269: main @ 0x000000000b27d81f E 18. ? @ 0x00007f1253d83d90 E 19. ? @ 0x00007f1253d83e40 E 20. _start @ 0x000000000b1a602e E (version 24.12.2.20221.altinityantalya (altinity build)) E (query: INSERT INTO insert_select_replicated_local SELECT * FROM s3Cluster( E 'first_shard', E 'http://minio1:9001/root/data/generated/*.csv', 'minio', 'minio123', 'CSV','a String, b UInt64' E ) SETTINGS parallel_distributed_insert_select=1;) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query DROP TABLE IF EXISTS insert_select_replicated_local ON CLUSTER 'first_shard' SYNC; on s0_0_0 Executing query DROP TABLE IF EXISTS insert_select_replicated_local ON CLUSTER 'first_shard' SYNC; on s0_0_0 Executing query CREATE TABLE insert_select_replicated_local ON CLUSTER 'first_shard' (a String, b UInt64) ENGINE=ReplicatedMergeTree('/clickhouse/tables/{shard}/insert_select_with_replicated', '{replica}') ORDER BY (a, b); on s0_0_0 Executing query CREATE TABLE insert_select_replicated_local ON CLUSTER 'first_shard' (a String, b UInt64) ENGINE=ReplicatedMergeTree('/clickhouse/tables/{shard}/insert_select_with_replicated', '{replica}') ORDER BY (a, b); on s0_0_0 Executing query SYSTEM STOP FETCHES; on s0_0_0 Executing query SYSTEM STOP FETCHES; on s0_0_0 Executing query SYSTEM STOP MERGES; on s0_0_0 Executing query SYSTEM STOP MERGES; on s0_0_0 Executing query SYSTEM STOP FETCHES; on s0_0_1 Executing query SYSTEM STOP FETCHES; on s0_0_1 Executing query SYSTEM STOP MERGES; on s0_0_1 Executing query SYSTEM STOP MERGES; on s0_0_1 Executing query INSERT INTO insert_select_replicated_local SELECT * FROM s3Cluster( 'first_shard', 'http://minio1:9001/root/data/generated/*.csv', 'minio', 'minio123', 'CSV','a String, b UInt64' ) SETTINGS parallel_distributed_insert_select=1; on s0_0_0 Executing query INSERT INTO insert_select_replicated_local SELECT * FROM s3Cluster( 'first_shard', 'http://minio1:9001/root/data/generated/*.csv', 'minio', 'minio123', 'CSV','a String, b UInt64' ) SETTINGS parallel_distributed_insert_select=1; on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-04-01 23:42:37 [ 643 ] DEBUG : Executing query DROP TABLE IF EXISTS insert_select_replicated_local ON CLUSTER 'first_shard' SYNC; on s0_0_0 (cluster.py:3677, query) 2025-04-01 23:42:37 [ 643 ] DEBUG : Executing query CREATE TABLE insert_select_replicated_local ON CLUSTER 'first_shard' (a String, b UInt64) ENGINE=ReplicatedMergeTree('/clickhouse/tables/{shard}/insert_select_with_replicated', '{replica}') ORDER BY (a, b); on s0_0_0 (cluster.py:3677, query) 2025-04-01 23:42:38 [ 643 ] DEBUG : Executing query SYSTEM STOP FETCHES; on s0_0_0 (cluster.py:3677, query) 2025-04-01 23:42:38 [ 643 ] DEBUG : Executing query SYSTEM STOP MERGES; on s0_0_0 (cluster.py:3677, query) 2025-04-01 23:42:38 [ 643 ] DEBUG : Executing query SYSTEM STOP FETCHES; on s0_0_1 (cluster.py:3677, query) 2025-04-01 23:42:39 [ 643 ] DEBUG : Executing query SYSTEM STOP MERGES; on s0_0_1 (cluster.py:3677, query) 2025-04-01 23:42:39 [ 643 ] DEBUG : Executing query INSERT INTO insert_select_replicated_local SELECT * FROM s3Cluster( 'first_shard', 'http://minio1:9001/root/data/generated/*.csv', 'minio', 'minio123', 'CSV','a String, b UInt64' ) SETTINGS parallel_distributed_insert_select=1; on s0_0_0 (cluster.py:3677, query) _______________________ test_distributed_s3_table_engine _______________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_distributed_s3_table_engine(started_cluster): node = started_cluster.instances["s0_0_0"] > resp_def = node.query( """ SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) """ ) test_s3_cluster/test.py:727: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3678: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 209, stderr: Code: 209. DB::NetException: Timeout: connect timed out: 172.16.8.11:9000 (172.16.8.11:9000, connection timeout 10000 ms). (SOCKET_TIMEOUT), Stack trace (when copying this message, always include the lines below): E E 0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x0000000038031254 E 1. ./build_docker/./src/Common/Exception.cpp:105: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001bc0ed05 E 2. ./src/Common/Exception.h:105: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String const&, long&&) @ 0x00000000307fa057 E 3. ./build_docker/./src/Client/Connection.cpp:321: DB::Connection::connect(DB::ConnectionTimeouts const&) @ 0x00000000307df744 E 4. ./build_docker/./src/Client/Connection.cpp:627: DB::Connection::getServerVersion(DB::ConnectionTimeouts const&, String&, unsigned long&, unsigned long&, unsigned long&, unsigned long&) @ 0x00000000307e6aaa E 5. ./build_docker/./programs/client/Client.cpp:479: DB::Client::connect() @ 0x000000001bff6862 E 6. ./build_docker/./programs/client/Client.cpp:375: DB::Client::main(std::vector> const&) @ 0x000000001bff437f E 7. ./build_docker/./base/poco/Util/src/Application.cpp:315: Poco::Util::Application::run() @ 0x00000000382716b7 E 8. ./build_docker/./programs/client/Client.cpp:1399: mainEntryClickHouseClient(int, char**) @ 0x000000001c010ec9 E 9. ./build_docker/./programs/main.cpp:269: main @ 0x000000000b27d81f E 10. ? @ 0x00007f8cf9f34d90 E 11. ? @ 0x00007f8cf9f34e40 E 12. _start @ 0x000000000b1a602e helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) on s0_0_0 Executing query SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-04-01 23:42:55 [ 643 ] DEBUG : Executing query SELECT * from s3Cluster( 'cluster_simple', 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) on s0_0_0 (cluster.py:3677, query) ____________________________ test_hive_partitioning ____________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_hive_partitioning(started_cluster): node = started_cluster.instances["s0_0_0"] for i in range(1, 5): > exists = node.query( f""" SELECT count() FROM s3('http://minio1:9001/root/data/hive/key={i}/*', 'minio', 'minio123', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV """ ) test_s3_cluster/test.py:794: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3678: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 209, stderr: Code: 209. DB::NetException: Timeout: connect timed out: 172.16.8.11:9000 (172.16.8.11:9000, connection timeout 10000 ms). (SOCKET_TIMEOUT), Stack trace (when copying this message, always include the lines below): E E 0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x0000000038031254 E 1. ./build_docker/./src/Common/Exception.cpp:105: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001bc0ed05 E 2. ./src/Common/Exception.h:105: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String const&, long&&) @ 0x00000000307fa057 E 3. ./build_docker/./src/Client/Connection.cpp:321: DB::Connection::connect(DB::ConnectionTimeouts const&) @ 0x00000000307df744 E 4. ./build_docker/./src/Client/Connection.cpp:627: DB::Connection::getServerVersion(DB::ConnectionTimeouts const&, String&, unsigned long&, unsigned long&, unsigned long&, unsigned long&) @ 0x00000000307e6aaa E 5. ./build_docker/./programs/client/Client.cpp:479: DB::Client::connect() @ 0x000000001bff6862 E 6. ./build_docker/./programs/client/Client.cpp:375: DB::Client::main(std::vector> const&) @ 0x000000001bff437f E 7. ./build_docker/./base/poco/Util/src/Application.cpp:315: Poco::Util::Application::run() @ 0x00000000382716b7 E 8. ./build_docker/./programs/client/Client.cpp:1399: mainEntryClickHouseClient(int, char**) @ 0x000000001c010ec9 E 9. ./build_docker/./programs/main.cpp:269: main @ 0x000000000b27d81f E 10. ? @ 0x00007f949883bd90 E 11. ? @ 0x00007f949883be40 E 12. _start @ 0x000000000b1a602e helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=1/*', 'minio', 'minio123', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=1/*', 'minio', 'minio123', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-04-01 23:43:06 [ 643 ] DEBUG : Executing query SELECT count() FROM s3('http://minio1:9001/root/data/hive/key=1/*', 'minio', 'minio123', 'Parquet', 'key Int32, value Int32') GROUP BY ALL FORMAT TSV on s0_0_0 (cluster.py:3677, query) ________ test_parallel_distributed_insert_select_with_schema_inference _________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_parallel_distributed_insert_select_with_schema_inference(started_cluster): node = started_cluster.instances["s0_0_0"] > node.query( """DROP TABLE IF EXISTS parallel_insert_select ON CLUSTER 'first_shard' SYNC;""" ) test_s3_cluster/test.py:436: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3678: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 209, stderr: Code: 209. DB::NetException: Timeout: connect timed out: 172.16.8.11:9000 (172.16.8.11:9000, connection timeout 10000 ms). (SOCKET_TIMEOUT), Stack trace (when copying this message, always include the lines below): E E 0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x0000000038031254 E 1. ./build_docker/./src/Common/Exception.cpp:105: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001bc0ed05 E 2. ./src/Common/Exception.h:105: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String const&, long&&) @ 0x00000000307fa057 E 3. ./build_docker/./src/Client/Connection.cpp:321: DB::Connection::connect(DB::ConnectionTimeouts const&) @ 0x00000000307df744 E 4. ./build_docker/./src/Client/Connection.cpp:627: DB::Connection::getServerVersion(DB::ConnectionTimeouts const&, String&, unsigned long&, unsigned long&, unsigned long&, unsigned long&) @ 0x00000000307e6aaa E 5. ./build_docker/./programs/client/Client.cpp:479: DB::Client::connect() @ 0x000000001bff6862 E 6. ./build_docker/./programs/client/Client.cpp:375: DB::Client::main(std::vector> const&) @ 0x000000001bff437f E 7. ./build_docker/./base/poco/Util/src/Application.cpp:315: Poco::Util::Application::run() @ 0x00000000382716b7 E 8. ./build_docker/./programs/client/Client.cpp:1399: mainEntryClickHouseClient(int, char**) @ 0x000000001c010ec9 E 9. ./build_docker/./programs/main.cpp:269: main @ 0x000000000b27d81f E 10. ? @ 0x00007f5be939fd90 E 11. ? @ 0x00007f5be939fe40 E 12. _start @ 0x000000000b1a602e helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query DROP TABLE IF EXISTS parallel_insert_select ON CLUSTER 'first_shard' SYNC; on s0_0_0 Executing query DROP TABLE IF EXISTS parallel_insert_select ON CLUSTER 'first_shard' SYNC; on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-04-01 23:43:16 [ 643 ] DEBUG : Executing query DROP TABLE IF EXISTS parallel_insert_select ON CLUSTER 'first_shard' SYNC; on s0_0_0 (cluster.py:3677, query) ______________________________ test_remote_hedged ______________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_remote_hedged(started_cluster): node = started_cluster.instances["s0_0_0"] > pure_s3 = node.query( """ SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) LIMIT 1 """ ) test_s3_cluster/test.py:672: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3678: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 209, stderr: Code: 209. DB::NetException: Timeout: connect timed out: 172.16.8.11:9000 (172.16.8.11:9000, connection timeout 10000 ms). (SOCKET_TIMEOUT), Stack trace (when copying this message, always include the lines below): E E 0. ./contrib/llvm-project/libcxx/include/exception:141: Poco::Exception::Exception(String const&, int) @ 0x0000000038031254 E 1. ./build_docker/./src/Common/Exception.cpp:105: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x000000001bc0ed05 E 2. ./src/Common/Exception.h:105: DB::NetException::NetException(int, FormatStringHelperImpl::type, std::type_identity::type, std::type_identity::type>, String&&, String const&, long&&) @ 0x00000000307fa057 E 3. ./build_docker/./src/Client/Connection.cpp:321: DB::Connection::connect(DB::ConnectionTimeouts const&) @ 0x00000000307df744 E 4. ./build_docker/./src/Client/Connection.cpp:627: DB::Connection::getServerVersion(DB::ConnectionTimeouts const&, String&, unsigned long&, unsigned long&, unsigned long&, unsigned long&) @ 0x00000000307e6aaa E 5. ./build_docker/./programs/client/Client.cpp:479: DB::Client::connect() @ 0x000000001bff6862 E 6. ./build_docker/./programs/client/Client.cpp:375: DB::Client::main(std::vector> const&) @ 0x000000001bff437f E 7. ./build_docker/./base/poco/Util/src/Application.cpp:315: Poco::Util::Application::run() @ 0x00000000382716b7 E 8. ./build_docker/./programs/client/Client.cpp:1399: mainEntryClickHouseClient(int, char**) @ 0x000000001c010ec9 E 9. ./build_docker/./programs/main.cpp:269: main @ 0x000000000b27d81f E 10. ? @ 0x00007fc788f30d90 E 11. ? @ 0x00007fc788f30e40 E 12. _start @ 0x000000000b1a602e helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) LIMIT 1 on s0_0_0 Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) LIMIT 1 on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-04-01 23:43:27 [ 643 ] DEBUG : Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) LIMIT 1 on s0_0_0 (cluster.py:3677, query) ____________________________ test_remote_no_hedged _____________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_remote_no_hedged(started_cluster): node = started_cluster.instances["s0_0_0"] > pure_s3 = node.query( """ SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) LIMIT 1 """ ) test_s3_cluster/test.py:699: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3678: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Net Exception: No route to host (172.16.8.11:9000). (NETWORK_ERROR) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) LIMIT 1 on s0_0_0 Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) LIMIT 1 on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-04-01 23:43:37 [ 643 ] DEBUG : Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) LIMIT 1 on s0_0_0 (cluster.py:3677, query) _______________________________ test_select_all ________________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_select_all(started_cluster): node = started_cluster.instances["s0_0_0"] > pure_s3 = node.query( """ SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon)""" ) test_s3_cluster/test.py:110: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3678: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Net Exception: No route to host (172.16.8.11:9000). (NETWORK_ERROR) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) on s0_0_0 Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-04-01 23:43:41 [ 643 ] DEBUG : Executing query SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ORDER BY (name, value, polygon) on s0_0_0 (cluster.py:3677, query) _________________________ test_skip_unavailable_shards _________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_skip_unavailable_shards(started_cluster): node = started_cluster.instances["s0_0_0"] > result = node.query( """ SELECT count(*) from s3Cluster( 'cluster_non_existent_port', 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS skip_unavailable_shards = 1 """ ) test_s3_cluster/test.py:315: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3678: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Net Exception: No route to host (172.16.8.11:9000). (NETWORK_ERROR) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT count(*) from s3Cluster( 'cluster_non_existent_port', 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS skip_unavailable_shards = 1 on s0_0_0 Executing query SELECT count(*) from s3Cluster( 'cluster_non_existent_port', 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS skip_unavailable_shards = 1 on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-04-01 23:43:44 [ 643 ] DEBUG : Executing query SELECT count(*) from s3Cluster( 'cluster_non_existent_port', 'http://minio1:9001/root/data/clickhouse/part1.csv', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') SETTINGS skip_unavailable_shards = 1 on s0_0_0 (cluster.py:3677, query) ________________________________ test_union_all ________________________________ [gw1] linux -- Python 3.10.12 /usr/bin/python3 started_cluster = def test_union_all(started_cluster): node = started_cluster.instances["s0_0_0"] > pure_s3 = node.query( """ SELECT * FROM ( SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ) ORDER BY (name, value, polygon) """ ) test_s3_cluster/test.py:204: _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ helpers/cluster.py:3678: in query return self.client.query( helpers/client.py:39: in wrap return func(self, *args, **kwargs) helpers/client.py:79: in query ).get_answer() _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ self = def get_answer(self): self.process.wait(timeout=DEFAULT_QUERY_TIMEOUT) self.stdout_file.seek(0) self.stderr_file.seek(0) stdout = self.stdout_file.read().decode("utf-8", errors="replace") stderr = self.stderr_file.read().decode("utf-8", errors="replace") if ( self.timer is not None and not self.process_finished_before_timeout and not self.ignore_error ): logging.debug(f"Timed out. Last stdout:{stdout}, stderr:{stderr}") raise QueryTimeoutExceedException("Client timed out!") if ( self.process.returncode != 0 or self.remove_trash_from_stderr(stderr) ) and not self.ignore_error: > raise QueryRuntimeException( "Client failed! Return code: {}, stderr: {}".format( self.process.returncode, stderr ), self.process.returncode, stderr, ) E helpers.client.QueryRuntimeException: Client failed! Return code: 210, stderr: Code: 210. DB::NetException: Net Exception: No route to host (172.16.8.11:9000). (NETWORK_ERROR) helpers/client.py:248: QueryRuntimeException ----------------------------- Captured stderr call ----------------------------- Executing query SELECT * FROM ( SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ) ORDER BY (name, value, polygon) on s0_0_0 Executing query SELECT * FROM ( SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ) ORDER BY (name, value, polygon) on s0_0_0 ------------------------------ Captured log call ------------------------------- 2025-04-01 23:43:47 [ 643 ] DEBUG : Executing query SELECT * FROM ( SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') UNION ALL SELECT * from s3( 'http://minio1:9001/root/data/{clickhouse,database}/*', 'minio', 'minio123', 'CSV', 'name String, value UInt32, polygon Array(Array(Tuple(Float64, Float64)))') ) ORDER BY (name, value, polygon) on s0_0_0 (cluster.py:3677, query) --------------------------- Captured stderr teardown --------------------------- Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/.env --project-name roottests3cluster-gw1 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_1_0/docker-compose.yml stop --timeout 20] Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/.env --project-name roottests3cluster-gw1 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_1_0/docker-compose.yml stop --timeout 20] Stderr: Container roottests3cluster-gw1-s0_0_0-1 Stopping Stderr: Container roottests3cluster-gw1-s0_0_0-1 Stopping Stderr: Container roottests3cluster-gw1-resolver-1 Stopping Stderr: Container roottests3cluster-gw1-resolver-1 Stopping Stderr: Container roottests3cluster-gw1-s0_0_1-1 Stopping Stderr: Container roottests3cluster-gw1-s0_0_1-1 Stopping Stderr: Container roottests3cluster-gw1-s0_1_0-1 Stopping Stderr: Container roottests3cluster-gw1-s0_1_0-1 Stopping Stderr: Container roottests3cluster-gw1-s0_0_0-1 Stopped Stderr: Container roottests3cluster-gw1-s0_0_0-1 Stopped Stderr: Container roottests3cluster-gw1-minio1-1 Stopping Stderr: Container roottests3cluster-gw1-minio1-1 Stopping Stderr: Container roottests3cluster-gw1-minio1-1 Stopped Stderr: Container roottests3cluster-gw1-minio1-1 Stopped Stderr: Container roottests3cluster-gw1-s0_1_0-1 Stopped Stderr: Container roottests3cluster-gw1-s0_1_0-1 Stopped Stderr: Container roottests3cluster-gw1-s0_0_1-1 Stopped Stderr: Container roottests3cluster-gw1-s0_0_1-1 Stopped Stderr: Container roottests3cluster-gw1-zoo3-1 Stopping Stderr: Container roottests3cluster-gw1-zoo3-1 Stopping Stderr: Container roottests3cluster-gw1-zoo1-1 Stopping Stderr: Container roottests3cluster-gw1-zoo1-1 Stopping Stderr: Container roottests3cluster-gw1-zoo2-1 Stopping Stderr: Container roottests3cluster-gw1-zoo2-1 Stopping Stderr: Container roottests3cluster-gw1-zoo1-1 Stopped Stderr: Container roottests3cluster-gw1-zoo1-1 Stopped Stderr: Container roottests3cluster-gw1-zoo2-1 Stopped Stderr: Container roottests3cluster-gw1-zoo2-1 Stopped Stderr: Container roottests3cluster-gw1-zoo3-1 Stopped Stderr: Container roottests3cluster-gw1-zoo3-1 Stopped Stderr: Container roottests3cluster-gw1-resolver-1 Stopped Stderr: Container roottests3cluster-gw1-resolver-1 Stopped Stderr: Container roottests3cluster-gw1-proxy2-1 Stopping Stderr: Container roottests3cluster-gw1-proxy2-1 Stopping Stderr: Container roottests3cluster-gw1-proxy1-1 Stopping Stderr: Container roottests3cluster-gw1-proxy1-1 Stopping Stderr: Container roottests3cluster-gw1-proxy2-1 Stopped Stderr: Container roottests3cluster-gw1-proxy2-1 Stopped Stderr: Container roottests3cluster-gw1-proxy1-1 Stopped Stderr: Container roottests3cluster-gw1-proxy1-1 Stopped Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_0/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_0/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_0/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_0/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_1_0/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_1_0/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_1_0/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_1_0/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/.env --project-name roottests3cluster-gw1 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_1_0/docker-compose.yml down --volumes] Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/.env --project-name roottests3cluster-gw1 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_1_0/docker-compose.yml down --volumes] Stderr: Container roottests3cluster-gw1-resolver-1 Stopping Stderr: Container roottests3cluster-gw1-resolver-1 Stopping Stderr: Container roottests3cluster-gw1-s0_1_0-1 Stopping Stderr: Container roottests3cluster-gw1-s0_1_0-1 Stopping Stderr: Container roottests3cluster-gw1-s0_0_1-1 Stopping Stderr: Container roottests3cluster-gw1-s0_0_1-1 Stopping Stderr: Container roottests3cluster-gw1-s0_0_0-1 Stopping Stderr: Container roottests3cluster-gw1-s0_0_0-1 Stopping Stderr: Container roottests3cluster-gw1-resolver-1 Stopped Stderr: Container roottests3cluster-gw1-resolver-1 Stopped Stderr: Container roottests3cluster-gw1-resolver-1 Removing Stderr: Container roottests3cluster-gw1-resolver-1 Removing Stderr: Container roottests3cluster-gw1-s0_0_0-1 Stopped Stderr: Container roottests3cluster-gw1-s0_0_0-1 Stopped Stderr: Container roottests3cluster-gw1-s0_0_0-1 Removing Stderr: Container roottests3cluster-gw1-s0_0_0-1 Removing Stderr: Container roottests3cluster-gw1-s0_1_0-1 Stopped Stderr: Container roottests3cluster-gw1-s0_1_0-1 Stopped Stderr: Container roottests3cluster-gw1-s0_1_0-1 Removing Stderr: Container roottests3cluster-gw1-s0_1_0-1 Removing Stderr: Container roottests3cluster-gw1-s0_0_1-1 Stopped Stderr: Container roottests3cluster-gw1-s0_0_1-1 Stopped Stderr: Container roottests3cluster-gw1-s0_0_1-1 Removing Stderr: Container roottests3cluster-gw1-s0_0_1-1 Removing Stderr: Container roottests3cluster-gw1-s0_0_1-1 Removed Stderr: Container roottests3cluster-gw1-s0_0_1-1 Removed Stderr: Container roottests3cluster-gw1-s0_0_0-1 Removed Stderr: Container roottests3cluster-gw1-s0_0_0-1 Removed Stderr: Container roottests3cluster-gw1-minio1-1 Stopping Stderr: Container roottests3cluster-gw1-minio1-1 Stopping Stderr: Container roottests3cluster-gw1-minio1-1 Stopped Stderr: Container roottests3cluster-gw1-minio1-1 Stopped Stderr: Container roottests3cluster-gw1-minio1-1 Removing Stderr: Container roottests3cluster-gw1-minio1-1 Removing Stderr: Container roottests3cluster-gw1-resolver-1 Removed Stderr: Container roottests3cluster-gw1-resolver-1 Removed Stderr: Container roottests3cluster-gw1-s0_1_0-1 Removed Stderr: Container roottests3cluster-gw1-s0_1_0-1 Removed Stderr: Container roottests3cluster-gw1-zoo1-1 Stopping Stderr: Container roottests3cluster-gw1-zoo1-1 Stopping Stderr: Container roottests3cluster-gw1-zoo2-1 Stopping Stderr: Container roottests3cluster-gw1-zoo2-1 Stopping Stderr: Container roottests3cluster-gw1-zoo3-1 Stopping Stderr: Container roottests3cluster-gw1-zoo3-1 Stopping Stderr: Container roottests3cluster-gw1-zoo1-1 Stopped Stderr: Container roottests3cluster-gw1-zoo1-1 Stopped Stderr: Container roottests3cluster-gw1-zoo1-1 Removing Stderr: Container roottests3cluster-gw1-zoo1-1 Removing Stderr: Container roottests3cluster-gw1-zoo2-1 Stopped Stderr: Container roottests3cluster-gw1-zoo2-1 Stopped Stderr: Container roottests3cluster-gw1-zoo2-1 Removing Stderr: Container roottests3cluster-gw1-zoo2-1 Removing Stderr: Container roottests3cluster-gw1-zoo3-1 Stopped Stderr: Container roottests3cluster-gw1-zoo3-1 Stopped Stderr: Container roottests3cluster-gw1-zoo3-1 Removing Stderr: Container roottests3cluster-gw1-zoo3-1 Removing Stderr: Container roottests3cluster-gw1-zoo1-1 Removed Stderr: Container roottests3cluster-gw1-zoo1-1 Removed Stderr: Container roottests3cluster-gw1-zoo2-1 Removed Stderr: Container roottests3cluster-gw1-zoo2-1 Removed Stderr: Container roottests3cluster-gw1-zoo3-1 Removed Stderr: Container roottests3cluster-gw1-zoo3-1 Removed Stderr: Container roottests3cluster-gw1-minio1-1 Removed Stderr: Container roottests3cluster-gw1-minio1-1 Removed Stderr: Container roottests3cluster-gw1-proxy1-1 Stopping Stderr: Container roottests3cluster-gw1-proxy1-1 Stopping Stderr: Container roottests3cluster-gw1-proxy2-1 Stopping Stderr: Container roottests3cluster-gw1-proxy2-1 Stopping Stderr: Container roottests3cluster-gw1-proxy2-1 Stopped Stderr: Container roottests3cluster-gw1-proxy2-1 Stopped Stderr: Container roottests3cluster-gw1-proxy2-1 Removing Stderr: Container roottests3cluster-gw1-proxy2-1 Removing Stderr: Container roottests3cluster-gw1-proxy1-1 Stopped Stderr: Container roottests3cluster-gw1-proxy1-1 Stopped Stderr: Container roottests3cluster-gw1-proxy1-1 Removing Stderr: Container roottests3cluster-gw1-proxy1-1 Removing Stderr: Container roottests3cluster-gw1-proxy1-1 Removed Stderr: Container roottests3cluster-gw1-proxy1-1 Removed Stderr: Container roottests3cluster-gw1-proxy2-1 Removed Stderr: Container roottests3cluster-gw1-proxy2-1 Removed Stderr: Volume roottests3cluster-gw1_data1-1 Removing Stderr: Volume roottests3cluster-gw1_data1-1 Removing Stderr: Network roottests3cluster-gw1_default Removing Stderr: Network roottests3cluster-gw1_default Removing Stderr: Volume roottests3cluster-gw1_data1-1 Removed Stderr: Volume roottests3cluster-gw1_data1-1 Removed Stderr: Network roottests3cluster-gw1_default Removed Stderr: Network roottests3cluster-gw1_default Removed Cleanup called Cleanup called Docker networks for project roottests3cluster-gw1 are NETWORK ID NAME DRIVER SCOPE Docker networks for project roottests3cluster-gw1 are NETWORK ID NAME DRIVER SCOPE Docker containers for project roottests3cluster-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker containers for project roottests3cluster-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES Docker volumes for project roottests3cluster-gw1 are DRIVER VOLUME NAME Docker volumes for project roottests3cluster-gw1 are DRIVER VOLUME NAME Command:[docker container list --all --filter name='^/roottests3cluster-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] Command:[docker container list --all --filter name='^/roottests3cluster-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] Unstopped containers: {} Unstopped containers: {} No running containers for project: roottests3cluster-gw1 No running containers for project: roottests3cluster-gw1 Trying to prune unused networks... Trying to prune unused networks... Trying to prune unused images... Trying to prune unused images... Command:[docker image prune -f] Command:[docker image prune -f] Stdout:Total reclaimed space: 0B Stdout:Total reclaimed space: 0B Images pruned Images pruned Trying to prune unused volumes... Trying to prune unused volumes... Command:[docker volume ls | wc -l] Command:[docker volume ls | wc -l] Stdout:5 Stdout:5 Command:[docker volume prune -f] Command:[docker volume prune -f] Stdout:Total reclaimed space: 0B Stdout:Total reclaimed space: 0B Volumes pruned: 5 Volumes pruned: 5 ---------------------------- Captured log teardown ----------------------------- 2025-04-01 23:43:50 [ 643 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/.env --project-name roottests3cluster-gw1 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_1_0/docker-compose.yml stop --timeout 20] (cluster.py:122, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-s0_0_0-1 Stopping (cluster.py:148, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-resolver-1 Stopping (cluster.py:148, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-s0_0_1-1 Stopping (cluster.py:148, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-s0_1_0-1 Stopping (cluster.py:148, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-s0_0_0-1 Stopped (cluster.py:148, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-minio1-1 Stopping (cluster.py:148, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-minio1-1 Stopped (cluster.py:148, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-s0_1_0-1 Stopped (cluster.py:148, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-s0_0_1-1 Stopped (cluster.py:148, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-zoo3-1 Stopping (cluster.py:148, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-zoo1-1 Stopping (cluster.py:148, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-zoo2-1 Stopping (cluster.py:148, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-zoo1-1 Stopped (cluster.py:148, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-zoo2-1 Stopped (cluster.py:148, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-zoo3-1 Stopped (cluster.py:148, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-resolver-1 Stopped (cluster.py:148, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-proxy2-1 Stopping (cluster.py:148, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-proxy1-1 Stopping (cluster.py:148, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-proxy2-1 Stopped (cluster.py:148, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-proxy1-1 Stopped (cluster.py:148, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_0/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_0/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:122, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_1/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_1/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:122, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Command:[bash -c [ -f /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_1_0/logs/stderr.log ] && zgrep -aH "==================" /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_1_0/logs/stderr.log* | ( [ -z "" ] && cat || grep -v "$" ) || true] (cluster.py:122, run_and_check) 2025-04-01 23:44:11 [ 643 ] DEBUG : Command:[docker compose --env-file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/.env --project-name roottests3cluster-gw1 --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_0/docker-compose.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_keeper.yml --file /ClickHouse/tests/integration/helpers/../../../tests/integration/compose/docker_compose_minio.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_0_1/docker-compose.yml --file /ClickHouse/tests/integration/test_s3_cluster/_instances-0-gw1/s0_1_0/docker-compose.yml down --volumes] (cluster.py:122, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-resolver-1 Stopping (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-s0_1_0-1 Stopping (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-s0_0_1-1 Stopping (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-s0_0_0-1 Stopping (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-resolver-1 Stopped (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-resolver-1 Removing (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-s0_0_0-1 Stopped (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-s0_0_0-1 Removing (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-s0_1_0-1 Stopped (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-s0_1_0-1 Removing (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-s0_0_1-1 Stopped (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-s0_0_1-1 Removing (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-s0_0_1-1 Removed (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-s0_0_0-1 Removed (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-minio1-1 Stopping (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-minio1-1 Stopped (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-minio1-1 Removing (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-resolver-1 Removed (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-s0_1_0-1 Removed (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-zoo1-1 Stopping (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-zoo2-1 Stopping (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-zoo3-1 Stopping (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-zoo1-1 Stopped (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-zoo1-1 Removing (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-zoo2-1 Stopped (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-zoo2-1 Removing (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-zoo3-1 Stopped (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-zoo3-1 Removing (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-zoo1-1 Removed (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-zoo2-1 Removed (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-zoo3-1 Removed (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-minio1-1 Removed (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-proxy1-1 Stopping (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-proxy2-1 Stopping (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-proxy2-1 Stopped (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-proxy2-1 Removing (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-proxy1-1 Stopped (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-proxy1-1 Removing (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-proxy1-1 Removed (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Container roottests3cluster-gw1-proxy2-1 Removed (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Volume roottests3cluster-gw1_data1-1 Removing (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Network roottests3cluster-gw1_default Removing (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Volume roottests3cluster-gw1_data1-1 Removed (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stderr: Network roottests3cluster-gw1_default Removed (cluster.py:148, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Cleanup called (cluster.py:894, cleanup) 2025-04-01 23:44:12 [ 643 ] DEBUG : Docker networks for project roottests3cluster-gw1 are NETWORK ID NAME DRIVER SCOPE (cluster.py:873, print_all_docker_pieces) 2025-04-01 23:44:12 [ 643 ] DEBUG : Docker containers for project roottests3cluster-gw1 are CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES (cluster.py:881, print_all_docker_pieces) 2025-04-01 23:44:12 [ 643 ] DEBUG : Docker volumes for project roottests3cluster-gw1 are DRIVER VOLUME NAME (cluster.py:889, print_all_docker_pieces) 2025-04-01 23:44:12 [ 643 ] DEBUG : Command:[docker container list --all --filter name='^/roottests3cluster-gw1-.*-1$' --format '{{.ID}}:{{.Names}}'] (cluster.py:122, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Unstopped containers: {} (cluster.py:908, cleanup) 2025-04-01 23:44:12 [ 643 ] DEBUG : No running containers for project: roottests3cluster-gw1 (cluster.py:922, cleanup) 2025-04-01 23:44:12 [ 643 ] DEBUG : Trying to prune unused networks... (cluster.py:928, cleanup) 2025-04-01 23:44:12 [ 643 ] DEBUG : Trying to prune unused images... (cluster.py:944, cleanup) 2025-04-01 23:44:12 [ 643 ] DEBUG : Command:[docker image prune -f] (cluster.py:122, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:146, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Images pruned (cluster.py:947, cleanup) 2025-04-01 23:44:12 [ 643 ] DEBUG : Trying to prune unused volumes... (cluster.py:953, cleanup) 2025-04-01 23:44:12 [ 643 ] DEBUG : Command:[docker volume ls | wc -l] (cluster.py:122, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stdout:5 (cluster.py:146, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Command:[docker volume prune -f] (cluster.py:122, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Stdout:Total reclaimed space: 0B (cluster.py:146, run_and_check) 2025-04-01 23:44:12 [ 643 ] DEBUG : Volumes pruned: 5 (cluster.py:958, cleanup) ============================== slowest durations =============================== 670.64s call test_replicated_merge_tree_s3_zero_copy/test.py::test_drop_table 300.79s call test_s3_aws_sdk_has_slightly_unreliable_behaviour/test.py::test_dataloss 48.79s call test_replicated_mutations/test.py::test_mutations_dont_prevent_merges[nodes1] 47.96s call test_replicated_mutations/test.py::test_mutations_dont_prevent_merges[nodes0] 45.21s call test_replicated_users/test.py::test_reload_zookeeper 42.21s call test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned 34.86s call test_replicated_mutations/test.py::test_mutations 26.43s setup test_refreshable_mat_view_replicated/test.py::test_alters[settings0-test_db-True-depends_on1-True-True] 23.59s setup test_replicated_mutations/test.py::test_mutations 23.39s setup test_replicated_table_attach/test.py::test_startup_with_small_bg_pool 23.12s setup test_s3_cluster/test.py::test_ambiguous_join 22.70s teardown test_replicated_merge_tree_s3_zero_copy/test.py::test_s3_check_restore 22.08s teardown test_s3_cluster/test.py::test_union_all 22.03s teardown test_s3_aws_sdk_has_slightly_unreliable_behaviour/test.py::test_dataloss 21.86s setup test_replicated_merge_tree_s3_zero_copy/test.py::test_drop_table 18.93s setup test_replicated_users/test.py::test_create_and_delete_replicated[QUOTA] 18.42s setup test_s3_aws_sdk_has_slightly_unreliable_behaviour/test.py::test_dataloss 18.37s setup test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids 18.24s call test_s3_cluster/test.py::test_distributed_insert_select_with_replicated 18.16s setup test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach 16.13s teardown test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach 11.85s teardown test_replicated_mutations/test.py::test_mutations_dont_prevent_merges[nodes1] 11.78s teardown test_replicated_merge_tree_encryption_codec/test.py::test_same_keys 10.45s call test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference 10.44s call test_s3_cluster/test.py::test_hive_partitioning 10.40s call test_s3_cluster/test.py::test_distributed_s3_table_engine 10.39s call test_s3_cluster/test.py::test_remote_hedged 7.22s call test_replicated_table_attach/test.py::test_startup_with_small_bg_pool 6.59s call test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces 5.96s call test_replicated_merge_tree_encryption_codec/test.py::test_same_keys 5.83s call test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-True-False] 5.65s call test_s3_cluster/test.py::test_cluster_format_detection 5.62s call test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids 4.48s call test_s3_cluster/test.py::test_cluster_default_expression 3.81s call test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-False-False] 3.49s call test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-False-True] 3.39s call test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-False-False] 3.26s teardown test_replicated_users/test.py::test_rename_replicated[USER] 3.22s call test_s3_cluster/test.py::test_remote_no_hedged 3.15s setup test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-False-False] 3.04s call test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-True-True] 2.99s call test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-True-True] 2.97s call test_s3_cluster/test.py::test_union_all 2.97s call test_s3_cluster/test.py::test_skip_unavailable_shards 2.97s call test_s3_cluster/test.py::test_select_all 2.97s call test_replicated_merge_tree_encryption_codec/test.py::test_different_keys 2.97s call test_refreshable_mat_view_replicated/test.py::test_alters[settings0-test_db-True-depends_on1-True-True] 2.95s call test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-True-False] 2.94s call test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-False-False] 2.90s teardown test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-True-True] 2.90s call test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-True-True] 2.75s call test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-True-True] 2.73s call test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-True-False] 2.69s call test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-False-True] 2.69s call test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-False-True] 2.69s call test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-True-False] 2.64s call test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-False-True] 2.63s call test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-False-False] 2.61s setup test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-False-True] 2.50s call test_s3_cluster/test.py::test_cluster_with_header 2.46s teardown test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned 2.46s call test_replicated_users/test.py::test_grant_revoke_replicated[Without_on_cluster] 2.41s setup test_refreshable_mat_view_replicated/test.py::test_append[False-False-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] 2.28s call test_refreshable_mat_view_replicated/test.py::test_append[True-False-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] 2.27s call test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[8192-12] 2.21s setup test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-True-False] 2.21s setup test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-True-False] 2.18s call test_refreshable_mat_view_replicated/test.py::test_append[False-False-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] 2.16s setup test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-True-True] 2.16s call test_s3_cluster/test.py::test_cluster_with_named_collection 2.13s setup test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-True-True] 2.11s call test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[0-16] 2.07s call test_refreshable_mat_view_replicated/test.py::test_append[True-False-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] 2.01s setup test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-False-True] 2.01s setup test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-False-False] 1.97s setup test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-False-False] 1.97s call test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[SETTINGS_PROFILE] 1.96s setup test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-False-False] 1.96s setup test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-False-True] 1.96s setup test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-True-False] 1.91s setup test_refreshable_mat_view_replicated/test.py::test_append[False-False-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] 1.91s setup test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-True-True] 1.91s setup test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-True-False] 1.91s setup test_refreshable_mat_view_replicated/test.py::test_append[True-True-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] 1.86s setup test_refreshable_mat_view_replicated/test.py::test_append[True-False-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] 1.86s setup test_refreshable_mat_view_replicated/test.py::test_long_query_cancel 1.86s setup test_refreshable_mat_view_replicated/test.py::test_append[False-True-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] 1.86s setup test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-False-False] 1.86s setup test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-False-True] 1.86s setup test_refreshable_mat_view_replicated/test.py::test_append[True-True-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] 1.83s call test_replicated_merge_tree_s3_zero_copy/test.py::test_s3_check_restore 1.81s setup test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-True-True] 1.81s setup test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-False-True] 1.81s setup test_refreshable_mat_view_replicated/test.py::test_append[False-True-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] 1.81s setup test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-False-True] 1.80s call test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[ROLE] 1.78s call test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[QUOTA] 1.76s setup test_refreshable_mat_view_replicated/test.py::test_append[True-False-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] 1.76s setup test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-True-False] 1.76s setup test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-True-True] 1.76s setup test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-True-False] 1.76s setup test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-True-True] 1.71s setup test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-False-False] 1.70s call test_replicated_users/test.py::test_grant_revoke_replicated[With_ignored_on_cluster] 1.57s call test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[USER] 1.55s call test_refreshable_mat_view_replicated/test.py::test_append[False-False-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] 1.50s call test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[ROW_POLICY] 1.48s call test_replicated_users/test.py::test_create_replicated_on_cluster[ROLE] 1.46s call test_refreshable_mat_view_replicated/test.py::test_append[True-True-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] 1.45s teardown test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[8192-12] 1.40s call test_refreshable_mat_view_replicated/test.py::test_append[True-True-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] 1.35s call test_refreshable_mat_view_replicated/test.py::test_append[False-True-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] 1.35s call test_refreshable_mat_view_replicated/test.py::test_append[False-True-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] 1.35s teardown test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[0-16] 1.29s call test_s3_cluster/test.py::test_count_macro 1.23s call test_replicated_users/test.py::test_create_replicated[QUOTA] 1.21s call test_replicated_users/test.py::test_create_replicated[ROW_POLICY] 1.16s call test_replicated_users/test.py::test_create_replicated[ROLE] 1.11s teardown test_replicated_merge_tree_s3_zero_copy/test.py::test_drop_table 1.10s call test_replicated_users/test.py::test_create_replicated[USER] 1.10s call test_replicated_users/test.py::test_create_replicated[SETTINGS_PROFILE] 1.01s call test_replicated_users/test.py::test_rename_replicated[SETTINGS_PROFILE] 0.96s setup test_refreshable_mat_view_replicated/test.py::test_query_retry 0.95s call test_replicated_users/test.py::test_create_and_delete_replicated[USER] 0.93s call test_s3_cluster/test.py::test_count 0.91s setup test_refreshable_mat_view_replicated/test.py::test_query_fail 0.87s call test_replicated_users/test.py::test_create_replicated_on_cluster[SETTINGS_PROFILE] 0.87s call test_replicated_users/test.py::test_create_replicated_on_cluster[USER] 0.86s call test_replicated_users/test.py::test_rename_replicated[USER] 0.81s call test_replicated_users/test.py::test_rename_replicated[ROLE] 0.76s call test_replicated_users/test.py::test_rename_replicated[ROW_POLICY] 0.76s call test_replicated_users/test.py::test_rename_replicated[QUOTA] 0.73s call test_replicated_users/test.py::test_create_replicated_on_cluster[ROW_POLICY] 0.72s call test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach 0.65s call test_replicated_users/test.py::test_create_replicated_on_cluster[QUOTA] 0.59s call test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[USER] 0.59s call test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[ROW_POLICY] 0.58s call test_replicated_users/test.py::test_create_and_delete_replicated[ROLE] 0.58s call test_replicated_users/test.py::test_create_and_delete_replicated[ROW_POLICY] 0.57s call test_s3_cluster/test.py::test_ambiguous_join 0.54s call test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[QUOTA] 0.52s call test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[SETTINGS_PROFILE] 0.50s call test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[ROLE] 0.44s call test_replicated_users/test.py::test_create_and_delete_replicated[SETTINGS_PROFILE] 0.38s call test_replicated_users/test.py::test_create_and_delete_replicated[QUOTA] 0.27s call test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-True-False] 0.22s call test_refreshable_mat_view_replicated/test.py::test_long_query_cancel 0.22s call test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-False-False] 0.22s call test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-True-False] 0.22s call test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-False-True] 0.22s call test_refreshable_mat_view_replicated/test.py::test_query_retry 0.22s call test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-False-True] 0.22s call test_refreshable_mat_view_replicated/test.py::test_query_fail 0.22s call test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-True-True] 0.22s call test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-False-False] 0.22s call test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-True-True] 0.01s teardown test_replicated_mutations/test.py::test_mutations 0.00s setup test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[0-16] 0.00s setup test_replicated_merge_tree_encryption_codec/test.py::test_different_keys 0.00s setup test_replicated_mutations/test.py::test_mutations_dont_prevent_merges[nodes0] 0.00s teardown test_replicated_users/test.py::test_create_replicated[QUOTA] 0.00s teardown test_replicated_users/test.py::test_create_replicated[ROLE] 0.00s teardown test_s3_cluster/test.py::test_cluster_format_detection 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_query_fail 0.00s setup test_replicated_users/test.py::test_create_replicated_on_cluster[ROLE] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_alters[settings0-test_db-True-depends_on1-True-True] 0.00s setup test_replicated_users/test.py::test_create_replicated_on_cluster[QUOTA] 0.00s teardown test_replicated_users/test.py::test_create_replicated[ROW_POLICY] 0.00s teardown test_s3_cluster/test.py::test_ambiguous_join 0.00s teardown test_replicated_mutations/test.py::test_mutations_dont_prevent_merges[nodes0] 0.00s setup test_replicated_users/test.py::test_create_replicated[QUOTA] 0.00s setup test_replicated_users/test.py::test_grant_revoke_replicated[With_ignored_on_cluster] 0.00s setup test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[8192-12] 0.00s teardown test_replicated_users/test.py::test_create_and_delete_replicated[QUOTA] 0.00s setup test_replicated_mutations/test.py::test_mutations_dont_prevent_merges[nodes1] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-True-True] 0.00s setup test_replicated_users/test.py::test_rename_replicated[ROW_POLICY] 0.00s setup test_replicated_users/test.py::test_create_replicated_on_cluster[ROW_POLICY] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-True-False] 0.00s setup test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[QUOTA] 0.00s setup test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[ROLE] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-False-False] 0.00s setup test_replicated_merge_tree_s3_zero_copy/test.py::test_s3_check_restore 0.00s setup test_replicated_users/test.py::test_create_and_delete_replicated[SETTINGS_PROFILE] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_append[True-False-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] 0.00s teardown test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids 0.00s setup test_replicated_users/test.py::test_reload_zookeeper 0.00s setup test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[USER] 0.00s teardown test_replicated_users/test.py::test_reload_zookeeper 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_append[False-False-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-False-False] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-False-False] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-True-False] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-False-True] 0.00s setup test_replicated_users/test.py::test_rename_replicated[QUOTA] 0.00s setup test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[QUOTA] 0.00s setup test_replicated_users/test.py::test_rename_replicated[SETTINGS_PROFILE] 0.00s setup test_s3_cluster/test.py::test_cluster_default_expression 0.00s teardown test_replicated_users/test.py::test_grant_revoke_replicated[Without_on_cluster] 0.00s teardown test_replicated_users/test.py::test_create_replicated_on_cluster[ROW_POLICY] 0.00s setup test_replicated_users/test.py::test_rename_replicated[ROLE] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_append[True-False-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] 0.00s setup test_replicated_users/test.py::test_create_and_delete_replicated[ROLE] 0.00s setup test_s3_cluster/test.py::test_cluster_with_header 0.00s teardown test_s3_cluster/test.py::test_hive_partitioning 0.00s setup test_replicated_users/test.py::test_create_replicated_on_cluster[SETTINGS_PROFILE] 0.00s setup test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference 0.00s teardown test_s3_cluster/test.py::test_distributed_insert_select_with_replicated 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-False-False] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-True-False] 0.00s setup test_replicated_users/test.py::test_create_replicated[ROLE] 0.00s setup test_replicated_users/test.py::test_create_and_delete_replicated[USER] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-True-True] 0.00s setup test_replicated_users/test.py::test_create_replicated[ROW_POLICY] 0.00s teardown test_replicated_users/test.py::test_create_replicated_on_cluster[ROLE] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-True-True] 0.00s teardown test_replicated_table_attach/test.py::test_startup_with_small_bg_pool 0.00s setup test_s3_cluster/test.py::test_distributed_s3_table_engine 0.00s setup test_replicated_users/test.py::test_rename_replicated[USER] 0.00s setup test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[SETTINGS_PROFILE] 0.00s teardown test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces 0.00s teardown test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[USER] 0.00s teardown test_replicated_users/test.py::test_rename_replicated[QUOTA] 0.00s setup test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_long_query_cancel 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-False-True] 0.00s setup test_replicated_users/test.py::test_create_and_delete_replicated[ROW_POLICY] 0.00s teardown test_s3_cluster/test.py::test_skip_unavailable_shards 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_append[False-True-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] 0.00s teardown test_replicated_users/test.py::test_create_and_delete_replicated[ROW_POLICY] 0.00s teardown test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[QUOTA] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-False-True] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-False-True] 0.00s setup test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[ROW_POLICY] 0.00s setup test_replicated_users/test.py::test_create_replicated[SETTINGS_PROFILE] 0.00s teardown test_s3_cluster/test.py::test_remote_hedged 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-False-True] 0.00s setup test_replicated_users/test.py::test_create_replicated[USER] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_append[False-False-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] 0.00s setup test_s3_cluster/test.py::test_remote_hedged 0.00s teardown test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[USER] 0.00s setup test_s3_cluster/test.py::test_count_macro 0.00s setup test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[ROLE] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_append[False-True-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_append[True-True-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] 0.00s setup test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[USER] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-False-False] 0.00s teardown test_replicated_users/test.py::test_rename_replicated[SETTINGS_PROFILE] 0.00s setup test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[ROW_POLICY] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-False-True] 0.00s setup test_replicated_users/test.py::test_grant_revoke_replicated[Without_on_cluster] 0.00s teardown test_replicated_users/test.py::test_rename_replicated[ROLE] 0.00s setup test_replicated_users/test.py::test_create_replicated_on_cluster[USER] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-True-True] 0.00s setup test_s3_cluster/test.py::test_distributed_insert_select_with_replicated 0.00s setup test_replicated_merge_tree_encryption_codec/test.py::test_same_keys 0.00s teardown test_s3_cluster/test.py::test_distributed_s3_table_engine 0.00s teardown test_replicated_users/test.py::test_create_replicated[USER] 0.00s setup test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[SETTINGS_PROFILE] 0.00s teardown test_replicated_users/test.py::test_create_replicated_on_cluster[USER] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-False-False] 0.00s setup test_s3_cluster/test.py::test_hive_partitioning 0.00s setup test_s3_cluster/test.py::test_union_all 0.00s teardown test_replicated_users/test.py::test_create_replicated[SETTINGS_PROFILE] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-True-False] 0.00s teardown test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[SETTINGS_PROFILE] 0.00s setup test_s3_cluster/test.py::test_remote_no_hedged 0.00s setup test_s3_cluster/test.py::test_select_all 0.00s teardown test_s3_cluster/test.py::test_remote_no_hedged 0.00s teardown test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference 0.00s teardown test_s3_cluster/test.py::test_select_all 0.00s setup test_s3_cluster/test.py::test_cluster_format_detection 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-True-True] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_append[True-True-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] 0.00s teardown test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[ROLE] 0.00s teardown test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[ROW_POLICY] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-True-False] 0.00s teardown test_replicated_users/test.py::test_rename_replicated[ROW_POLICY] 0.00s teardown test_s3_cluster/test.py::test_cluster_with_header 0.00s setup test_s3_cluster/test.py::test_skip_unavailable_shards 0.00s teardown test_s3_cluster/test.py::test_cluster_with_named_collection 0.00s teardown test_replicated_users/test.py::test_create_replicated_on_cluster[QUOTA] 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_query_retry 0.00s setup test_s3_cluster/test.py::test_count 0.00s setup test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned 0.00s setup test_s3_cluster/test.py::test_cluster_with_named_collection 0.00s teardown test_s3_cluster/test.py::test_count 0.00s teardown test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-True-False] 0.00s teardown test_s3_cluster/test.py::test_count_macro 0.00s teardown test_replicated_users/test.py::test_create_and_delete_replicated[ROLE] 0.00s teardown test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[SETTINGS_PROFILE] 0.00s teardown test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[QUOTA] 0.00s teardown test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[ROLE] 0.00s teardown test_replicated_users/test.py::test_create_and_delete_replicated[USER] 0.00s teardown test_replicated_users/test.py::test_grant_revoke_replicated[With_ignored_on_cluster] 0.00s teardown test_replicated_users/test.py::test_create_replicated_on_cluster[SETTINGS_PROFILE] 0.00s teardown test_replicated_users/test.py::test_create_and_delete_replicated[SETTINGS_PROFILE] 0.00s teardown test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[ROW_POLICY] 0.00s teardown test_replicated_merge_tree_encryption_codec/test.py::test_different_keys 0.00s teardown test_s3_cluster/test.py::test_cluster_default_expression =========================== short test summary info ============================ FAILED test_s3_cluster/test.py::test_distributed_insert_select_with_replicated FAILED test_s3_cluster/test.py::test_distributed_s3_table_engine - helpers.cl... FAILED test_s3_cluster/test.py::test_hive_partitioning - helpers.client.Query... FAILED test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference FAILED test_s3_cluster/test.py::test_remote_hedged - helpers.client.QueryRunt... FAILED test_s3_cluster/test.py::test_remote_no_hedged - helpers.client.QueryR... FAILED test_s3_cluster/test.py::test_select_all - helpers.client.QueryRuntime... FAILED test_s3_cluster/test.py::test_skip_unavailable_shards - helpers.client... FAILED test_s3_cluster/test.py::test_union_all - helpers.client.QueryRuntimeE... PASSED test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach PASSED test_replicated_users/test.py::test_create_and_delete_replicated[QUOTA] PASSED test_replicated_users/test.py::test_create_and_delete_replicated[ROLE] PASSED test_replicated_users/test.py::test_create_and_delete_replicated[ROW_POLICY] PASSED test_replicated_users/test.py::test_create_and_delete_replicated[SETTINGS_PROFILE] PASSED test_replicated_users/test.py::test_create_and_delete_replicated[USER] PASSED test_replicated_users/test.py::test_create_replicated[QUOTA] PASSED test_s3_cluster/test.py::test_ambiguous_join PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids PASSED test_replicated_users/test.py::test_create_replicated[ROLE] PASSED test_replicated_users/test.py::test_create_replicated[ROW_POLICY] PASSED test_replicated_users/test.py::test_create_replicated[SETTINGS_PROFILE] PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_keys PASSED test_replicated_users/test.py::test_create_replicated[USER] PASSED test_s3_cluster/test.py::test_cluster_default_expression PASSED test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[QUOTA] PASSED test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[ROLE] PASSED test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[ROW_POLICY] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings0-test_db-True-depends_on1-True-True] PASSED test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[SETTINGS_PROFILE] PASSED test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[USER] PASSED test_replicated_table_attach/test.py::test_startup_with_small_bg_pool PASSED test_replicated_users/test.py::test_create_replicated_on_cluster[QUOTA] PASSED test_replicated_users/test.py::test_create_replicated_on_cluster[ROLE] PASSED test_replicated_users/test.py::test_create_replicated_on_cluster[ROW_POLICY] PASSED test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces PASSED test_s3_cluster/test.py::test_cluster_format_detection PASSED test_replicated_users/test.py::test_create_replicated_on_cluster[SETTINGS_PROFILE] PASSED test_replicated_users/test.py::test_create_replicated_on_cluster[USER] PASSED test_s3_cluster/test.py::test_cluster_with_header PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-False-False] PASSED test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[QUOTA] PASSED test_s3_cluster/test.py::test_cluster_with_named_collection PASSED test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[ROLE] PASSED test_s3_cluster/test.py::test_count PASSED test_replicated_merge_tree_encryption_codec/test.py::test_same_keys PASSED test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[ROW_POLICY] PASSED test_s3_cluster/test.py::test_count_macro PASSED test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[SETTINGS_PROFILE] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-False-True] PASSED test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[USER] PASSED test_replicated_users/test.py::test_grant_revoke_replicated[With_ignored_on_cluster] PASSED test_replicated_users/test.py::test_grant_revoke_replicated[Without_on_cluster] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-True-False] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-True-True] PASSED test_replicated_mutations/test.py::test_mutations PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-False-False] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-False-True] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-True-False] PASSED test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-True-True] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-False-False] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-False-True] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-True-False] PASSED test_replicated_users/test.py::test_reload_zookeeper PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-True-True] PASSED test_replicated_users/test.py::test_rename_replicated[QUOTA] PASSED test_replicated_users/test.py::test_rename_replicated[ROLE] PASSED test_replicated_users/test.py::test_rename_replicated[ROW_POLICY] PASSED test_replicated_users/test.py::test_rename_replicated[SETTINGS_PROFILE] PASSED test_replicated_users/test.py::test_rename_replicated[USER] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-False-False] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-False-True] PASSED test_replicated_mutations/test.py::test_mutations_dont_prevent_merges[nodes0] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-True-False] PASSED test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-True-True] PASSED test_refreshable_mat_view_replicated/test.py::test_append[False-False-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] PASSED test_refreshable_mat_view_replicated/test.py::test_append[False-False-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] PASSED test_refreshable_mat_view_replicated/test.py::test_append[False-True-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] PASSED test_refreshable_mat_view_replicated/test.py::test_append[False-True-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] PASSED test_refreshable_mat_view_replicated/test.py::test_append[True-False-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] PASSED test_refreshable_mat_view_replicated/test.py::test_append[True-False-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] PASSED test_refreshable_mat_view_replicated/test.py::test_append[True-True-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0] PASSED test_refreshable_mat_view_replicated/test.py::test_append[True-True-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0] PASSED test_replicated_mutations/test.py::test_mutations_dont_prevent_merges[nodes1] PASSED test_s3_aws_sdk_has_slightly_unreliable_behaviour/test.py::test_dataloss PASSED test_replicated_merge_tree_s3_zero_copy/test.py::test_drop_table PASSED test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[0-16] PASSED test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[8192-12] PASSED test_replicated_merge_tree_s3_zero_copy/test.py::test_s3_check_restore SKIPPED [1] test_refreshable_mat_view_replicated/test.py:516: Disabled for sanitizers SKIPPED [1] test_refreshable_mat_view_replicated/test.py:569: Disabled for sanitizers SKIPPED [1] test_refreshable_mat_view_replicated/test.py:600: Disabled for sanitizers SKIPPED [8] test_refreshable_mat_view_replicated/test.py:374: Disabled for sanitizers ============= 9 failed, 80 passed, 11 skipped in 727.80s (0:12:07) ============= Traceback (most recent call last): File "/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration/./runner", line 528, in subprocess.check_call(cmd, shell=True) File "/usr/lib/python3.10/subprocess.py", line 369, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command 'docker run --rm --name clickhouse_integration_tests_3ycre4 --privileged --dns-search='.' --memory=30709035008 --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-odbc-bridge:/clickhouse-odbc-bridge --volume=/home/ubuntu/_work/_temp/test/build/clickhouse:/clickhouse --volume=/home/ubuntu/_work/_temp/test/build/clickhouse-library-bridge:/clickhouse-library-bridge --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/programs/server:/clickhouse-config --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/tests/integration:/ClickHouse/tests/integration --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/backupview:/ClickHouse/utils/backupview --volume=/home/ubuntu/_work/ClickHouse/ClickHouse/utils/grpc-client/pb2:/ClickHouse/utils/grpc-client/pb2 --volume=/run:/run/host:ro --volume=clickhouse_integration_tests_volume:/var/lib/docker -e DOCKER_DOTNET_CLIENT_TAG=11de0b29a15d -e DOCKER_HELPER_TAG=5dc43a6382f0 -e DOCKER_BASE_TAG=6712d5cc610d -e DOCKER_KERBEROS_KDC_TAG=9391ecdee8d7 -e DOCKER_MYSQL_GOLANG_CLIENT_TAG=9bec2a638e6e -e DOCKER_MYSQL_JAVA_CLIENT_TAG=766bff31cfe4 -e DOCKER_MYSQL_JS_CLIENT_TAG=41ba7c2ec2a1 -e DOCKER_MYSQL_PHP_CLIENT_TAG=88be89c1e3b6 -e DOCKER_NGINX_DAV_TAG=b55ac9cd7519 -e DOCKER_POSTGRESQL_JAVA_CLIENT_TAG=a4eff5c7f4d6 -e DOCKER_PYTHON_BOTTLE_TAG=caad4729259e -e DOCKER_CLIENT_TIMEOUT=300 -e COMPOSE_HTTP_TIMEOUT=600 -e PYTHONUNBUFFERED=1 -e PYTEST_ADDOPTS="--dist=loadfile -n 10 -rfEps --run-id=0 --color=no --durations=0 'test_refreshable_mat_view_replicated/test.py::test_alters[settings0-test_db-True-depends_on1-True-True]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-False-False]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-False-True]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-True-False]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-None-True-True]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-False-False]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-False-True]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-True-False]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-False-depends_on1-True-True]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-False-False]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-False-True]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-True-False]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-None-True-True]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-False-False]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-False-True]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-True-False]' 'test_refreshable_mat_view_replicated/test.py::test_alters[settings1-test_db-True-depends_on1-True-True]' 'test_refreshable_mat_view_replicated/test.py::test_append[False-False-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0]' 'test_refreshable_mat_view_replicated/test.py::test_append[False-False-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0]' 'test_refreshable_mat_view_replicated/test.py::test_append[False-True-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0]' 'test_refreshable_mat_view_replicated/test.py::test_append[False-True-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0]' 'test_refreshable_mat_view_replicated/test.py::test_append[True-False-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0]' 'test_refreshable_mat_view_replicated/test.py::test_append[True-False-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0]' 'test_refreshable_mat_view_replicated/test.py::test_append[True-True-SELECT now() as a, b as b FROM src1 SETTINGS insert_deduplicate=0]' 'test_refreshable_mat_view_replicated/test.py::test_append[True-True-SELECT now() as a, number as b FROM numbers(2) SETTINGS insert_deduplicate=0]' test_refreshable_mat_view_replicated/test.py::test_long_query_cancel test_refreshable_mat_view_replicated/test.py::test_query_fail test_refreshable_mat_view_replicated/test.py::test_query_retry 'test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-False-False]' 'test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-False-True]' 'test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-True-False]' 'test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause0-True-True]' 'test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-False-False]' 'test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-False-True]' 'test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-True-False]' 'test_refreshable_mat_view_replicated/test.py::test_real_wait_refresh[to_clause1-True-True]' test_replicated_merge_tree_encryption_codec/test.py::test_different_current_key_ids test_replicated_merge_tree_encryption_codec/test.py::test_different_keys test_replicated_merge_tree_encryption_codec/test.py::test_different_nonces test_replicated_merge_tree_encryption_codec/test.py::test_same_keys test_replicated_merge_tree_s3_zero_copy/test.py::test_drop_table 'test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[0-16]' 'test_replicated_merge_tree_s3_zero_copy/test.py::test_insert_select_replicated[8192-12]' test_replicated_merge_tree_s3_zero_copy/test.py::test_s3_check_restore test_replicated_mutations/test.py::test_mutations 'test_replicated_mutations/test.py::test_mutations_dont_prevent_merges[nodes0]' 'test_replicated_mutations/test.py::test_mutations_dont_prevent_merges[nodes1]' test_replicated_parse_zk_metadata/test.py::test_replicated_engine_parse_metadata_on_attach test_replicated_table_attach/test.py::test_startup_with_small_bg_pool test_replicated_table_attach/test.py::test_startup_with_small_bg_pool_partitioned 'test_replicated_users/test.py::test_create_and_delete_replicated[QUOTA]' 'test_replicated_users/test.py::test_create_and_delete_replicated[ROLE]' 'test_replicated_users/test.py::test_create_and_delete_replicated[ROW_POLICY]' 'test_replicated_users/test.py::test_create_and_delete_replicated[SETTINGS_PROFILE]' 'test_replicated_users/test.py::test_create_and_delete_replicated[USER]' 'test_replicated_users/test.py::test_create_replicated[QUOTA]' 'test_replicated_users/test.py::test_create_replicated[ROLE]' 'test_replicated_users/test.py::test_create_replicated[ROW_POLICY]' 'test_replicated_users/test.py::test_create_replicated[SETTINGS_PROFILE]' 'test_replicated_users/test.py::test_create_replicated[USER]' 'test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[QUOTA]' 'test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[ROLE]' 'test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[ROW_POLICY]' 'test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[SETTINGS_PROFILE]' 'test_replicated_users/test.py::test_create_replicated_if_not_exists_on_cluster[USER]' 'test_replicated_users/test.py::test_create_replicated_on_cluster[QUOTA]' 'test_replicated_users/test.py::test_create_replicated_on_cluster[ROLE]' 'test_replicated_users/test.py::test_create_replicated_on_cluster[ROW_POLICY]' 'test_replicated_users/test.py::test_create_replicated_on_cluster[SETTINGS_PROFILE]' 'test_replicated_users/test.py::test_create_replicated_on_cluster[USER]' 'test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[QUOTA]' 'test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[ROLE]' 'test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[ROW_POLICY]' 'test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[SETTINGS_PROFILE]' 'test_replicated_users/test.py::test_create_replicated_on_cluster_ignore[USER]' 'test_replicated_users/test.py::test_grant_revoke_replicated[With_ignored_on_cluster]' 'test_replicated_users/test.py::test_grant_revoke_replicated[Without_on_cluster]' test_replicated_users/test.py::test_reload_zookeeper 'test_replicated_users/test.py::test_rename_replicated[QUOTA]' 'test_replicated_users/test.py::test_rename_replicated[ROLE]' 'test_replicated_users/test.py::test_rename_replicated[ROW_POLICY]' 'test_replicated_users/test.py::test_rename_replicated[SETTINGS_PROFILE]' 'test_replicated_users/test.py::test_rename_replicated[USER]' test_s3_aws_sdk_has_slightly_unreliable_behaviour/test.py::test_dataloss test_s3_cluster/test.py::test_ambiguous_join test_s3_cluster/test.py::test_cluster_default_expression test_s3_cluster/test.py::test_cluster_format_detection test_s3_cluster/test.py::test_cluster_with_header test_s3_cluster/test.py::test_cluster_with_named_collection test_s3_cluster/test.py::test_count test_s3_cluster/test.py::test_count_macro test_s3_cluster/test.py::test_distributed_insert_select_with_replicated test_s3_cluster/test.py::test_distributed_s3_table_engine test_s3_cluster/test.py::test_hive_partitioning test_s3_cluster/test.py::test_parallel_distributed_insert_select_with_schema_inference test_s3_cluster/test.py::test_remote_hedged test_s3_cluster/test.py::test_remote_no_hedged test_s3_cluster/test.py::test_select_all test_s3_cluster/test.py::test_skip_unavailable_shards test_s3_cluster/test.py::test_union_all -vvv" altinityinfra/integration-tests-runner:cd6390247eca ' returned non-zero exit status 1.